It works on my notebook! Theory versus practice in Data Science

It works on my notebook! Theory versus practice in Data Science


When you start on the road to a
Data Driven company, you begin to understand how to use the tools that the current market offers such as artificial intelligence or Internet of Things. You hire data scientists who can solve your business problems without first asking yourself WHAT you need to do artificial intelligence and HOW you plan to do it.

The answer to the first question is simple, we need data. Now, the situation starts to get complicated when we try to answer our second question.

Everything seems perfect, but where do I have this data, what format does it take, is it easy to access, how often can I access it, is it complete, without errors, without null records, how long have I had this data, how long have I had this data? And assuming all this is solved, how easy is it to develop a Machine Learning model and implement it?

There are many questions, but there are also many answers depending on the problem to be solved.

When we talk about Data Science, we are not talking about a tool, skill or method, but more like a scientific approach that uses statistical theory, applied mathematics and computer tools to process large amounts of data. Data science is a detailed process that mainly involves preprocessing, analysis, visualization and prediction.

We all know that Data Science is a very powerful scientific approach, with all kinds of interesting applications. However, it is also well known that in Data Science there is a big gap between theory and practice: when it comes to theory, we know everything, but we don’t know how to apply it in real life.

For this reason it is important to prioritize when working with data. This list may change depending on the company, but most of them agree on many of these points.

Step 1 – Define the business problem

This first step is fundamental, and requires much more of the human factor for the understanding of the problem to be solved, the agreement of criteria for the definition of the objectives, scope and timeframe, than of the system itself that will be used as a means to reach them.

Surely the data scientist has many ways to solve a problem, but the one who must set the course of the solution must be the one who knows the business. Interaction and teamwork are essential.

Step 2 – Data acquisition

This step is perhaps where we find the biggest difference between theory and practice. In theory, when we want to make a machine learning model, all we need to do is download a dataset from sites such as Kaggle or Github and we will have clear, neat and well described information. In practice, sometimes the sources can be:

  1. Very varied: which would take a previous work of ETL’s, modeling, etc.
  2. Poorly described or without description: Without having a clear description of what variable we are working with we do not know what we have and if it can help us to solve our problem.
  3. With erroneous data / null records: As it is often said in Data Science, Garbage in / Garbage out.
  4. Unknown: In a sectorized company where the data are within the area that worked with them, the opportunity to combine them or use them for other business purposes may be lost.
  5. With restricted access: Depending on the data security standards within the company, accessing data often becomes a titanic task and involves a bureaucratic process that is difficult to measure over time.

These and many problems with data sources can be solved with proper data governance and fundamentally a very well communicated organization.

Step 3 – Data preparation

This step involves data cleansing and data transformation, data cleansing is the most time consuming as it involves handling many complex scenarios such as inconsistent data types, misspelled attributes, missing values and duplicates. Then, in data transformation, we have to modify the data based on the defined mapping rules.

Step 4 – Exploratory Data Analysis

With the help of Exploratory Data Analysis we define and refine the selection of variables to be used for the development of our model. It is important to always keep in mind the solution we want to target.

Step 5 – Data modeling

The main activity of a data science project is known as data modeling. In this step, we repeatedly apply machine learning techniques of type strength such as KNN, decision trees, Naive Bayes, etc. to the data so that we can identify the model that best fits the business requirement. We train the model on the training dataset and test it to select the best performing model.

Step 6 – Visualization and communication

This point is perhaps the most relevant of all because we can have the best data extraction and transformation process, the best trained Machine Learning model, but if we do not know how to visualize it, explain it, communicate it and give value to the business, all the previous work will not matter much. It is essential to reinforce soft skills at this point to know how to reach stakeholders.

Step 7 – Implementation and maintenance

And finally, in this step, the data scientist implements and maintains the model, tests the selected model in a pre-production environment before implementing it in the production environment, which is the best practice. After implementing it, we have to get real-time analytics and monitor and maintain the performance of the project.

As you will see, there is a huge difference between what we study (Theory) that practically starts and ends in a local Notebook versus what is needed to carry out the whole process in real life (Practice). It is for this reason that it is often overwhelming and sometimes frustrating to try to work with data and generate results.

That is why Macrotest #DataLab helps you all the way with our end-to-end solution so that you have a complete understanding of the tools, methodologies and processes.

How are agile cells composed?

How are composed the agile cells


Recall that in the previous note we talked about agile work cells. This is a way of organizing work similar to that of a living cell, reflecting its dynamism. In this environment, interdisciplinary and collaborative work is promoted, where each member of the cell contributes to the achievement of the objectives, always focused on the customer. 

With respect to the roles that make up this type of cell, they generally represent different areas and disciplines, all of which are necessary to respond to the problems that led to the formation of the cell. In particular, a Product Owner is usually included, who maintains the vision of the final product and transmits it to all those involved, seeking to ensure that in each sprint the product is being adapted to the needs of the end user. He/she must also have interpersonal skills since he/she is in charge of interacting with all the actors involved in the project. 

The Scrum Master is another role present in agile cells and acts as a facilitator, accompanying the process and seeking to ensure that the principles of the agile methodology are followed. His mission is to ensure the availability of the development team, eliminating obstacles that may arise and potentially affect the delivery of the product under the agreed terms. His skills at a strategic level, his ability to manage teamwork, and above all his problem-solving skills are important. 

Of course, the Development Team, whose mission is fundamentally to develop the product, working in small teams where information is shared openly and a collaborative spirit reigns, cannot be missing. There may also be other roles, such as Front End Developer, Back End Developer, Machine Learning specialists, UX Designer, among others.

Despite the many advantages of agile work cells, there can also be certain drawbacks during their development. Some of them are mainly associated with the lack of experience of the team, when these cells are entirely made up of junior members. In this case, the creation of agile work cells mentored by a figure with a high level of seniority is a very interesting tool. In this way, even when the client hires a cell made up of junior profiles and the fees reflect this, having a senior profile in charge of acting as a mentor raises the quality of the deliverable product. This is because the mentor guides the group of junior members and also reviews their deliverables. 

Another disadvantage to be avoided when working in agile cells is the lack of quality controls. In this sense, the eagerness to reduce development times to a minimum can lead to quality failures and ultimately to problems with the client. The role of a mentor with a high level of seniority can also contribute to avoiding this type of problems by guiding and reviewing the work developed by the team. These mentors provide their knowledge, tools, training and coaching to make the most of the team’s potential, but the most important thing is the experience they bring to a group of junior professionals. Their skills include not only technical skills, which are fundamental, but also interpersonal skills such as empathy, communication, patience and problem solving. 

At Macrotest we have agile work cells and also mentored cells, where projects developed by junior professionals go to the next level thanks to the senior mentoring process, pursuing the highest quality standards. 

Contact us for more information! 

What does agile cell work consist of?

Working with agile cells

 

Nowadays, the development of projects through the organization in agile cells is widely spread. As its name suggests, this form of work organization seeks to resemble the behavior of a living cell, in constant change and dynamism. This model is in a way in opposition to the traditional model based on hierarchies, which no longer works successfully in a highly changing environment and with increasingly demanding end customers.

Following this living cell model, in an agile work cell each team member retains his or her autonomy but is in constant interrelation with the other members. In this sense, while each member has a very clear role, this role is complemented by the roles and tasks of the others.

This type of work cells are generally composed of a small number of interdisciplinary professionals who share certain characteristics. Among them, proactivity and creativity stand out, considering that these cells work in a self-managed manner and are free to find solutions in the process of responding to a specific need. It is also key to build good interpersonal relationships, seeking to make the work work work as a whole. Likewise, collaborative work is the essence of this type of cells, where each member contributes to the achievement of the objectives with a focus on the needs of the client, who is always in contact with the project. 

In short, it is about generating an environment of innovation and total openness, pursuing the ultimate goal that the whole is more than the sum of its parts. This way of working seeks to speed up software development times, recognizing the importance of human resources even above technologies.

When is it recommended to outsource by hiring an agile development team?

Agile cells are very useful when knowledge and experience are needed. There are different ways of working according to each project, and the cell can be managed by the client itself, by the company that provides the technology professionals or even jointly. Agile cells are also very useful when looking for an efficient end-to-end development, since they seek to minimize deviations from the initial estimate. 

Outsourcing by hiring agile cells of IT professionals, allows the client to focus on the core of their business, reducing the technical barriers of new projects, investing only in what is needed according to the different types of profiles required. 

In Macrotest we have work cells available for your projects, integrated by multidisciplinary and modular specialized teams that are integrated into the structure of your organizations to develop the technological solutions you require. If you want to know more about our agile cells service, please contact us and we will gladly provide you with all the necessary information. At your disposal you will find all our experience, highly qualified resources, the highest quality standards and our framework that has been evolving for years. All this expertise is at the service of our clients, seeking to optimize their business processes and ensure quality and efficiency in each of the projects entrusted to us.

In the next article we will tell you about a particular type of agile cell, which are the mentored cells. We look forward to sharing this next note with you!

 

Technology trends… what’s the latest?

Technology trends

Technology is advancing continuously, at an unprecedented pace, and its application is taking place in the most diverse sectors. Agriculture and health are two of the most dynamic sectors where technology is constantly advancing. Below, we review some of the most relevant developments in terms of technological trends.

In the agricultural sector in particular, technological innovations are of such magnitude that they have led to the emergence of new concepts such as “green revolution” or “agriculture 4.0”. An example can be environmental monitoring through electronic sensors installed on crops and the possibility of automating irrigation systems. Another of the most frequent technological applications is the use of drones that, through remote sensing, make it possible to obtain images and videos remotely. The application of artificial intelligence to the management of a large volume of data from agricultural machinery, for example, is also key, as is the Internet of Things (IoT), which allows real-time information to be obtained and facilitates traceability through the installation of devices such as chips. It also highlights the automation of machines and the use of GPS in tractors, allowing to know the exact measurements and distances in crops. The application of Big Data also entails a series of important benefits in the agricultural sector, such as obtaining information on weather conditions, the state of the soil, the crops planted, the amount of inputs needed according to the environment, and identifying anomalies in crops, such as pests or diseases. 

In short, the application of these different technologies in agriculture allows not only to improve yields or increase efficiency in the use of resources, but also to anticipate and identify certain risks. For example, at Macrotest we have the Agroint solution for intelligent crop management, minimizing the impact of pests, diseases and weeds through monitoring and geolocated records from cell phones.

The health sector is not lagging behind in terms of new technological trends. In recent years, technology applied to healthcare has not only focused its efforts on saving lives, but also on making daily life more comfortable.  

Among the main technologies is the use of virtual reality in medical training, involving realistic scenarios such as surgery or emergencies. Once again, the application of artificial intelligence appears as fundamental, whether to monitor patients, assist surgeries by means of automated devices, or even predict the situation of high-risk patients. 

Another trend is access to remote healthcare, for example through online chat platforms available 24 hours a day through chatbots that apply artificial intelligence, or even medical consultations by video call. There are also implantable and ingestible sensors that facilitate the diagnosis, treatment and monitoring of certain diseases through the Internet of Things.

Mobile applications can also be mentioned, which contribute to improving the healthcare system through the prevention, control and treatment of numerous diseases, including monitoring the follow-up of patients with diseases such as diabetes or epilepsy. Meanwhile, Cloud Computing applied to the healthcare sector makes it possible to speed up access to patients’ clinical data, as well as to improve communication between healthcare centers. These trends as a whole currently imply benefits both for healthcare professionals and for the users of the healthcare system. 

In this line in Macrotest we have developed a solution for patients who suffered a stroke which allows the realization of their rehabilitation from home, based on years of research, development and innovation. 

We are waiting for you to deepen these applications that promote the development of the agro and health sectors.

What is cloud computing?

Cloud computing

Cloud computing is a technology that allows remote access to software, file storage and data processing using the Internet, which is an excellent alternative to running on a personal computer or local server.

Beyond its definition, what is really important is that cloud computing has radically transformed business models and business strategies, making it possible to accelerate innovation processes and reduce costs, not only for large companies but also for small and medium-sized enterprises.

Cloud computing provides companies with greater flexibility in relation to their data, being able to access it from anywhere and at any time, which is ideal not only for companies with headquarters in different locations around the world, but also to facilitate collaborative work between individuals in the same location.

In short, cloud computing is an intelligent way to take advantage of the benefits of the hyperconnection provided by the Internet, helping to eliminate hardware maintenance costs and software upgrades, and allowing the integration of all company information in a single place.

And what does it bring to companies?

The main advantage is that it makes it possible to have all the company’s information updated in real time. In this sense, the cloud allows control of the data, being able to choose which users have access to what type of data and with what level of permissions. Another great advantage is the virtually unlimited storage capacity, which frees up the computer and improves its performance.

It also helps to generate a much more fluid communication between departments, avoiding errors, duplications, inefficiencies and wasted time. Especially if employees work remotely or in different departments, cloud computing favors collaborative work, allowing documents and information to be shared in real time. Another of its great advantages is especially valued in unforeseen situations of data loss, since data recovery is one of the key aspects. Through cloud computing, data will always be available, even if we delete it by mistake. In addition, it is a multiplatform technology, which means that it can be accessed from a laptop, tablet, smartphone or computer just by having access to the Internet.

In short, cloud computing provides businesses of any size, the ability to access a set of applications designed to manage all information related to different business processes remotely and integrated.

In Macrotest we have experience in the cloud, mainly Azure and Aws, with all our implementations we accompany the progressive implementation of this type of technological strategies, to improve and add value in the different business processes. Contact us!

What does it mean to be a partner in R&D&I projects?

partner innovation

The development of the most innovative projects in terms of technological applications is highly complex and usually requires the collaboration of different actors who contribute their particular skills and knowledge to achieve the most disruptive results.

This collaboration scheme involves the joint work of two or more organizations in specific projects, an alliance usually known as “partners” or technology partners.

Partners are usually highly innovative companies, and can be start-ups, small and medium-sized enterprises, large companies, and even multinationals. In some projects, in addition to the participation of companies, partnerships with research centers and universities can also be made. This ecosystem is ideal for exploring and exploiting new ideas for innovation, contributing to the development of innovative services and solutions of the highest quality and reliability.

In general, these are companies that have highly qualified teams and are committed to research, development, innovation and digital transformation, providing complementary capabilities and technologies for the purposes of the project they develop under this partner scheme, which allows accelerating project times and introducing great improvements in the results. In many cases, they even work with a global network that combines the work of actors distributed around the world.

The philosophy underlying these collaborative projects is to jointly build business solutions, generating synergies and making the most of what each of the parties can contribute in terms of capabilities, experience, technology, networking, financing, and other resources. These projects generally focus on research, development and innovation (R&D&I) applicable to the most diverse sectors of activity.

This way of working occurs when projects could not be developed by a single company, necessarily requiring cooperation between different actors, or when collaboration between companies is clearly beneficial, facilitating and enhancing such projects.

In Macrotest, throughout our journey investing in innovation, development and research, we have generated valuable alliances with different partners, which has allowed us to achieve great results based on the synergy of teamwork.

We invite all those companies and institutions that want to explore joint work opportunities to contact us to continue growing together!

How to manage a project successfully?

The crisis generated by COVID-19 has accelerated the digital revolution in all business sectors. The irruption of information technology in all aspects of our lives has changed the paradigm of business management.

This new stage will have many challenges for companies and this will involve embarking on new projects. For this, work teams will need solid foundations to achieve the objectives they set themselves, which implies methodical and organized work plans. 

What to take into account when organizing a project? 

  1. Determine the scope: first of all, define the desired and result and what is needed to achieve it. For example, if you want to increase the amount of sales by a certain percentage (the final result), the actions to be implemented will be sales promotions, marketing and commercial strategies, content that attracts customers, etc.  
  2. Create a project outline: once the scope has been established, it is time to visually project it to the team. This can be done through a document available to everyone, showing an overview and the status of the project, where the progress of the project will be added. 
  3. Planning: each action or step of the project should be detailed here, along with its tentative completion dates, making it more understandable for the entire team.
  4. Assigning roles: when assigning roles and responsibilities, not only create a list of collaborators for each objective, but also take into account how to distribute resources in order to be efficient and obtain optimal results. 
  5. Review the project with the team: finally, it is important that everyone works together in the project management, determining if all tasks were assigned correctly and with the necessary time according to the collaborator. 

It is true that the management of a project is not simple, it is necessary to consider the time variable as well as to adjust to the assigned budget and the expected quality. Even in many cases, in the initial stages of a project, it is detected that in order to be carried out under the expected standards, the company must make large investments in infrastructure or personnel, causing the project to be abandoned due to lack of resources.

In these cases the solution is given by the outsourcing of services. In Macrotest, we have partners and a synergistic team with extensive experience in management and implementation of IT projects, both in the Latin American and European market.

If you are starting or accelerating your project and want to join our line of partners, do not hesitate to contact us!

Why Outsourcing Services?

Outsourcing is the name given in practice to the outsourcing of responsibilities in the IT sector, such as hiring teams for the development of projects or the execution of specific tasks.

The hiring of this type of services allows companies to focus on the business without having to consume human and financial resources in areas that are not strategic to their core business.

Benefits of Outsourcing Services 

The idea of outsourcing gives the company the freedom to dedicate itself exclusively to its core business, hiring specialized companies to perform tasks that are necessary for the development of the company – but that are not part of the business. This brings multiple benefits:

  1. Increased performance and productivity by avoiding unnecessary distractions. The work is done automatically through the contracting of services managed by a company specialized in the area. In this way, task completion times are reduced, executing the work faster and with higher quality.
  2. Savings in human and financial resources. This also leads to cost savings, since outsourcing saves the costs associated with infrastructure, hiring workers, etc.
  3. Focus on activities that are strategic and decisive for the business. They will stop investing their resources in areas that are not decisive for the business activity.
  4. Better margins. By saving costs, financial risks are reduced. By saving on human resources, labor risks are reduced. And so on and so forth. The company will not need to make large investments in technological infrastructure, so it will be able to allocate it to other new projects.
  5. Higher quality. External suppliers will have the specific knowledge for the activity to be developed, ensuring quality results.
  6. Transformation of fixed cost into variable cost. This means that technological investments are reduced by hiring experts, obtaining a more detailed cost forecast.
  7. Greater business flexibility. Companies will be much better prepared to face market changes, since they can hire services when they need them.
  8. Guaranteed access to the latest technological advances. Suppliers have the latest technology, so the company can take advantage of it without having to make a large investment. These suppliers are the perfect bridge between the company and technological innovation.
  9. Increased competitiveness. Companies improve their strategic position in the market and position themselves ahead of the competition.

At Macrotest we have a strategically managed team of professionals with extensive experience, which will avoid taking unnecessary risks or making large investments in both infrastructure and staffing. With numerous successful projects already in the market, the work synergy achieved with our clients can be seen in the way we meet the objectives we set together in each task performed.

If you want to carry out a project and you are looking for a partner to make it grow, through its development capacity, do not hesitate to contact us!

Translated with www.DeepL.com/Translator (free version)