About the client
Our client in the public sector, telecommunications domain, faced challenges in processing and analyzing large volumes of data from Telegram chats. They required a targeted search based on specific locations, keywords, and issues related to utility services. Telegram doesn’t support this functionality.
Out of the box, this messenger supports search only within one chat, by exact text, and by single text input. Built-in Telegram functionality would cause the client to waste too much time on manual searches. This had a significant impact on their ability to make swift, informed business decisions.
We suggested an efficient solution for extracting meaningful insights from vast amounts of data across multiple chats. To execute this concept, we formed a bootcamp startup development team under the supervision of our experienced experts to design an ETL service that focuses on classifying and aggregating messages and their authors within chats based on specific locations and keywords, including a search either by exact text or a search pattern. All this functionality had to be displayed on a user-friendly web interface.
Challenges regarding the management of a business
To build an effective business management system, our client needed an efficient solution to have access to up-to-date analytics, analyze vast amounts of information, and make well-informed decisions based on relevant data. The company’s founder in the utility services defined the following key challenges in attaining these objectives:
- Little budget for digital transformation creates tension between operational needs and long-term data infrastructure investments, which is a key concern for the VP of Finance, who must balance cost control with the need for scalable solutions.
- Data puddle with fragmented, unstructured data, lacking centralized oversight.
- The need to integrate spanerse data sources.
- Chaotic data storage, making analytics and decision-making more difficult.
- No in-house specialized technical team was available to begin the development process promptly.
Finding a team for a short-term project that can quickly spane into the full cycle development process for a specific client request is a challenging task. NetLS has found a way to bridge this gap by cultivating in-house teams through intensive training programs focused on particular domains.
Our NetLS specialists designed a practical learning course that covered all the steps in the development process of ETL for the data warehouse in digestible chunks of information, tailored for entry-level developers. C# .NET Full-Stack Bootcamp 2024: Two proof-of-concept ideas in one month helped our students acquire skill levels that were up to par for the project.
We approached the client’s case like Startup Development, adapting the team accordingly so that they could fulfill all the requirements and deliver the product on time with quality in mind.
How the work was done
Our discovery team studied this request and made a well-designed plan for solution realization. We designed a detailed Product Requirements Document with the necessary product features, technical specifications, and user requirements. Furthermore, the strategic roadmap kept us on track:
- Discovery stage. At this stage, the team analyzed users’ requirements and developed the core concept.
- Development stage. The team ensured that all technical solutions were aligned with the project’s architecture to maintain system stability and integrity.
- Delivery stage. The final stage included deployment and final testing.
Planning the work
Automation was the core idea of our solution. We aimed to streamline the process of analyzing data without the need for additional personnel. This reduced manual effort and ensured faster data output regarding emerging issues. AI/ML integration with Databricks enabled the solution to detect the problem and generate forecasts related to alerts and complaints.
Another critical aspect of our solution was structuring and visualizing information in real time. This feature allowed our client to have relevant analytics and examine operational efficiency and service delivery through historical changes. In addition, having all the information on the interactive dashboard makes it easier for the business to make more well-informed and timely decisions.
To address these tasks, we launched the second annual C# .NET Full-Stack internship, where our expert mentors, middle developers and tech supervisors, guided the team through the development process and share experience.
NetLS has a strong reputation that has helped us attract hundreds of applications from high-performing students with technical backgrounds. We selected top candidates from engineering and mathematics institutes and began the intensive course to prepare interns for further work.
Our unique training program enabled us to build a dedicated development team with a solid grasp of object-oriented programming, back-end development with C#, MS SQL, front-end technologies (HTML, CSS, JavaScript, TypeScript, Bootstrap, Angular, SCSS), and cloud services such as Azure AD, Azure Functions, SharePoint, and Power BI. Thus, our students could start working on the development process that included:
- Developing a web service that can process large volumes of data from the messenger and regularly update it.
- Ensuring backend stability and performance.
- Implementing a user-friendly and fast interface.
- Building analytical reports using Power Bi extract transform and load capabilities to deliver precise and actionable insights.
- Storing files and images in a structured format on SharePoint.
- Reliable authentication and access control.
- Modular architecture for flexible component replacement and scalability.
The first version of the solution architecture required the use of Data Puddle as a data processing layer. In the second version, the system had to be integrated with Data Lake in conjunction with Data Puddle to aggregate and store data efficiently.
In both iterations, we needed to ensure a well-designed ETL process for Data Puddle, which properly categorized, stored, and prepared incoming data from Telegram for analysis without loss or duplication.
Professional supervision
We formed a robust working group to help our dedicated development team complete the assigned tasks:
- C# software developer acted as the project lead, overseeing progress and assigning tasks to the students.
- Project manager was involved in the launches and was responsible for timely task completion.
- The technical lead acted as an adviser, overseeing code quality.
Our top engineers bring years of expertise in developing high-quality products, which is a cornerstone for launching a successful startup. Moreover, by collaborating with our robust team, the company’s in-house developers can significantly benefit from this invaluable experience, gaining insights into the best industry practices.
Thanks to exceptional methodologies, expert mentors, and a well-structured training system, the team was able to develop ETL data pipelines smoothly and effectively.
Development process
For website development, our working group chose .NET Core as the principal framework for the back end. This robust framework enabled us to ensure the stability and high performance of the service, which requires processing large volumes of data from the messenger and regularly updating it. Furthermore, data was updated every hour through Azure Functions.
The front end was developed using Angular, with routing for smooth navigation. Messages and keywords were stored in Cosmos DB, allowing efficient handling of large datasets. Quick and precise insights were displayed in analytical reports generated in Power BI. A REST API helped to retrieve and process data.
The students also implemented file and image storage in SharePoint, organized with a clear directory structure. Entity Framework was used for user authentication and access control.
Our dedicated team implemented data pipelines for ETL that ensured a reliable flow and transformation of information from the messaging platform into a structured format. That made the solution suitable for business analysis and decision-making.
By the end of the development process, our team released two versions:
In the first version, the main page includes the following:
- Data grids display information about keywords, messages, synonyms, authors, and chats. Each contains several columns with information and supports filtering by various criteria, including period, author, status, and others.
- Data operations that allow users to add, edit, or delete records, such as keywords, messages, synonyms, and chats. For deletions, a text field is provided to specify the reason for the deletion.
- Pagination is implemented for all grids to facilitate navigation when dealing with large amounts of data.
- Actions on records come with buttons for deleting, updating, and saving data, as well as the ability to return to previous versions during editing.
- Information in grids displays various types of data, including the number of descendants, messages for keywords, message status, synonym types, author information, and chat records.
- The file and image handling feature ensures that profile pictures, as well as files related to messages and authors, are stored properly.
In the second version, adding Data Lake as a layer between Telegram and our solution enabled the following functions:
- The Data Lake retrieves all messages from the selected Telegram chats, stores them for 24 hours in any format, and then deletes them.
- A restricted amount of 20 Telegram updates per hour per phone number.
- Our solution retrieves messages based on keywords, not directly from Telegram, but from Data Lake.
Results
The best practices and methodologies helped us form an effective team and guide interns with a structured program and schedule. Ultimately, the final release fully meets the customer’s requirements.
Our client achieved enhanced efficiency and effectiveness thanks to the following features in our solution:
- Detailed search through issues. Efficient search capabilities allow the client to filter problems related to utility services by location (country, city, region, and street), keywords, and type of issues.
- Structured data. Our practical solution can parse messages and save them in a structured format. Thus, the client can track historical changes and improvements in the performance of utility services.
- Real-time data visualization. Relevant data displayed on the interactive dashboards allows for more accurate decisions based on current, up-to-now data.
Our Startup Development services have become the most affordable and efficient work model for our client. It’s the best option for entrepreneurs with an idea but no expert tech team. In this cooperation model, we take full responsibility for the entire development cycle and deliver the expected results fast. Our robust team of developers teaches the client’s team to work with new technologies and how to use the best software development practices.
If the existing team needs additional support or faster delivery, we bring in experienced specialists who can quickly launch the project, streamline releases, and initiate a working unit economics model.
If you are interested in our expertise in this case, we’ll provide you with an additional presentation on this project. Please complete the contact form, and our team will be in touch with you shortly to discuss how we can assist your business!