Data Competencies: a Quantitative Analysis of Market Demands Via Text Mining on Linkedin

Diego Garcia Acedo; Daniela Cristina Silva Oliveira

DOI: 10.22167/2675-6528-202602845

Article derived from a Final Course Work (TCC), with content based on the student’s original work and adapted to the editorial format of the E&S Magazine with the support of the ResumeAI tool, an artificial intelligence solution developed by Instituto Pecege for textual synthesis and organization.

Summary

The advancement of digitalization and the exponential growth of data production have transformed the job market, increasing the demand for professionals in the areas of Data Engineering, Data Science, and Business Intelligence, whose responsibilities and skills had poorly defined boundaries. In this context, the study aimed to identify and analyze the most demanded technical and behavioral skills in job postings on the LinkedIn platform, seeking to map patterns, recurring characteristics, and differences between occupational profiles and seniority levels. A quantitative and descriptive approach was adopted, using web scraping techniques for automated collection of public job posting information. Subsequently, the data were organized, categorized, and analyzed using descriptive statistics and text mining, with emphasis on n-gram analysis. The results revealed that SQL (47%), Python (42%), cloud computing (40%), and communication (38%) were the most recurrent skills. It was observed that the Data Engineer profile focused on infrastructure and data processing, Data Scientist on analytical, statistical, and computational skills, and Business Intelligence on visualization, descriptive analysis, and business interface. The complexity of skills and the required seniority varied significantly among profiles. It was concluded that the findings contribute to the understanding of the transformations in the data field, offering subsidies for professional training and the alignment of recruitment and talent development practices with the effective demands of the market.

Keywords: Professional skills; Technical and behavioral skills; Data job market; Text mining; Web scraping.

1. Introduction

The advancement of digitalization and the exponential growth of data production have transformed the global landscape, generating an unprecedented amount of information. In this context, the ability to manage and process this data strategically has become a crucial factor for competitive advantage in the corporate environment (VICARIO; COLEMAN, 2019).

This process drove a profound transformation in the world of work, with the growing demand for professionals in the areas of technology and data. However, new occupations emerged whose definitions and boundaries are still ambiguous. There is a recurring difficulty in delimiting the functions and responsibilities of these specialists, with some focusing on machine learning, others on data infrastructure engineering, and a third strand acting more strategically, close to management and business intelligence (Davenport and Patil, 2012; Fayyad and Hamutcu, 2020; WING, 2021).

To understand these dynamics, it is fundamental to distinguish the concepts of work, occupation, and employment, which, although related, have distinct meanings in economic and sociological analyses (ANTUNES, 2009; CASTEL, 1998; POCHMANN, 2012). Work refers to human activity in the production of goods, services, and knowledge (MARX, 2013; ANTUNES, 2009). Occupation, in turn, designates the type of activity performed, such as data scientist or data engineer (MENEGASSO, 2000). Employment, on the other hand, represents the institutionalized form of labor relations, which, according to Castel (1998), has seen its modalities of professional insertion expand, combining flexibility and technical specialization (POCHMANN, 2012). This understanding is essential for analyzing the configuration of occupations in the data sector and the evolution of required skills (ANTUNES, 2018; CASTEL, 2009).

The technological labor market in Brazil showed significant growth of 95% between 2012 and 2022, according to data from the Annual Social Information Report (RAIS) analyzed by the Federation of Commerce of Goods, Services and Tourism of the State of São Paulo (FecomercioSP) (AGÊNCIA BRASIL, 2024). This period was marked by profound technological transformations, such as the expansion of the internet, the evolution of cloud computing, artificial intelligence, and the intensive use of big data, which redefined business models and drove the creation of new digital occupations (DAVENPORT; HARRIS, 2017; FAYYAD; HAMUTCU, 2020). Despite economic fluctuations and the COVID-19 pandemic (ANTUNES, 2020), the accumulated growth highlights the consolidation of this field as a strategic vector for competitiveness and innovation.

With the insertion of the internet and digital devices in various sectors, data professionals need to demonstrate not only proficiency in technical competencies but also a diverse set of interpersonal and cognitive skills to deal with the complexity of modern data analysis. The interdisciplinary nature of data science requires the integration of knowledge in statistics, programming, machine learning, and data management, combined with socioemotional skills such as communication, critical thinking, collaboration, and adaptability (DUBEY; GUNASEKARAN, 2015; CIMATTI, 2016; FAYYAD; HAMUTCU, 2020). Technical competencies, or hard skills, correspond to practical and specific knowledge, while socioemotional competencies, or soft skills, encompass behavioral and interpersonal aspects (OECD, 2019; WORLD ECONOMIC FORUM, 2020; LAKER; POWELL, 2011; CIMATTI, 2016). The valuation of these interpersonal skills has been reinforced in several studies, especially in the areas of education and business (ANDREWS; HIGSON, 2008; SUCCI; CANOVI, 2020).

Technical proficiency is an essential factor for performance in data science, demanding mastery of programming languages such as Python and R, in addition to techniques for statistical modeling, data manipulation, and visualization (SMITH et al., 2018). Python, a high-level language, is valued for its simple syntax and libraries for data manipulation and machine learning (MCKINNEY, 2018; VAN ROSSUM; DRAKE, 2009). R, in turn, is designed for statistical analysis and visualization, with an ecosystem of packages for modeling and regression (WICKHAM; GROLEMUND, 2017). Both are fundamental tools for data scientists and analysts, being prerequisites for technical performance in these occupations (SMITH et al., 2018; FAYYAD; HAMUTCU, 2020; ZAHARIA et al., 2016).

The ambiguity in the data market is not limited to which skills a professional should possess, but also to the level of knowledge required for each role. This discussion about the need for advanced degrees or specialized certifications, and the very definition of a “senior scientist,” reinforces the importance of understanding how technical and socio-emotional competencies are distributed and valued in various data occupations (FAYYAD; HAMUTCU, 2020). Given this gap, this study is justified by the need to offer subsidies for professional training and for aligning recruitment and talent development practices with the effective demands of the market, and aims to identify and analyze the most demanded technical and behavioral competencies in job openings in data published on the LinkedIn platform, seeking to map patterns, recurring characteristics, and differences between occupational profiles and seniority levels.

2. Material and Methods

The research adopted a quantitative and descriptive approach, aiming to identify and analyze the technical and behavioral competencies demanded in job openings in the data area. For this purpose, four main methodological procedures were employed: web scraping for data collection, descriptive analysis, assessment of the association between skills, and text mining.

The unit of analysis consisted of 2,621 job postings in data, published on the LinkedIn Jobs platform. Data collection was carried out in the last week of March 2026, focusing on opportunities for Data Engineer, Data Scientist, and Business Intelligence profiles, with a location filter for Brazil.

Data extraction was performed using the web scraping technique, which consists of the automated collection of public information from the internet (BOEGERSHAUSEN, 2022). For this purpose, a Python script was developed, configured to navigate page by page and extract information such as job title, company, location, and work modality (in-person, remote, or hybrid).

The development environment was configured using the Anaconda distribution, ensuring reproducible code execution. The Python libraries employed included Selenium, for web navigation automation, Chromedriver-autoinstaller, for managing the browser driver, and BeautifulSoup (bs4), for converting the HTML content of job descriptions into structured text.

Additionally, the Pandas library was used to structure the collected data in tabular format and for exporting to Excel files. Auxiliary libraries such as re, urllib.parse, pickle, datetime, time, and random were employed for text processing, URL manipulation, and random pause control, integrating web automation, textual processing, and data organization.

The script’s logic involved authenticating on the LinkedIn platform with session persistence via cookies and dynamically constructing search URLs. The results were loaded incrementally, simulating page scrolling to display new job openings. Each identified job opening was accessed individually to extract structural information and the complete description, which was converted from HTML to text and segmented.

To circumvent the limitations of the LinkedIn platform, such as dynamic loading and automation detection, random pauses were implemented between script actions. Multiple alternative selectors were also used to locate page elements, exception handling for loading failures, and periodic checkpoints to save collected data, ensuring the robustness of the process.

The descriptive analysis of the data was performed to organize, synthesize, and compare the collected datasets (REIS; REIS, 2002). This step allowed for observing the distribution of vacancies by location, companies, and qualifications, as well as identifying patterns in different vacancies of the same entry.

The delimitation of the set of skills to be considered in the analysis aimed to identify the most recurrent competencies in the job descriptions. These competencies were organized into thematic categories, including Database, Visualization and Reporting, Big Data, Programming, Statistics and Modeling, Cloud Computing, and Behavioral Competencies (Soft Skills).

Text mining was applied to interpret the textual content and identify recurring terms and expressions in the job descriptions. This approach transformed the textual content into quantitative variables, which were analyzed using statistical and computational methods, with emphasis on n-gram analysis (SCHONLAU et al., 2017).

During the development of the work, Artificial Intelligence (AI) was used as a support tool for organization, textual revision, and linguistic enhancement of the content. The language assistant ChatGPT (version GPT-5, OpenAI, 2025) was employed exclusively for bibliographic and textual review, language standardization, and citation formatting, in addition to supporting the development of Python codes for data extraction and treatment. The use of AI was conducted under the supervision and validation of the author, in accordance with the Manual of Good Practices for the Use of Artificial Intelligence of USP/ESALQ (2024), preserving the intellectual authorship and the researcher’s full responsibility.

3. Results and Discussion

The analysis of the results was conducted from a sample of 2,621 vacancies related to the areas of Data Engineering, data science, and Business Intelligence, collected on the LinkedIn platform. Initial data processing included textual standardization, removal of irrelevant terms, and normalization of expressions, aiming to reduce noise and ensure greater analytical consistency. This process allowed for a detailed characterization of the labor market demands in data, aligning with the objective of identifying the most required competencies.

The distribution of professional terms in the sample revealed a remarkable balance between the analyzed profiles. Approximately 35% of the positions were concentrated in the Data Scientist profile, 33% corresponded to Business Intelligence positions, and 32% to the Data Engineer profile. This homogeneous distribution suggests that the data job market in Brazil equally demands the three professional axes, indicating a diversity of roles and specializations within the data ecosystem.

Geographic information, after standardization and association with federative units, showed a strong concentration of vacancies in the state of São Paulo, which represented approximately 55% of the sample. This centrality of São Paulo in the Brazilian data market reflects its position as an economic and technological hub. Additionally, the analysis incorporated vacancies classified as “Brazil (Remote)” and “Latin America (Remote)”, which, although not linked to a specific location, significantly increased the volume of opportunities, with about 875 vacancies for remote Brazil and 69 for remote Latin America, highlighting the relevance of remote work in the sector.

When analyzing the geographic distribution by municipality, disregarding fully remote positions, a concentration of opportunities was observed in large urban centers. São Paulo stood out with 1,247 positions, far above other cities. Following São Paulo were Barueri with 72 positions, Osasco with 69, Rio de Janeiro with 63, Campinas with 44, and Belo Horizonte with 41. This pattern reinforces the centrality of the São Paulo capital and its immediate surroundings in the data ecosystem, while also revealing the presence of other relevant centers in different Brazilian states.

The distribution of work modalities by city revealed distinct patterns. In São Paulo, the hybrid model predominated, followed by in-person, with a smaller participation of remote work. In cities like Barueri, Osasco, Campinas, and Rio de Janeiro, the concentration in hybrid and in-person positions was even more pronounced, suggesting that, outside the capital, companies tend to require more physical interaction. This behavior may be associated with more centralized corporate structures or dependence on local operations.

In contrast, cities such as Belo Horizonte, Recife, Curitiba, Salvador, Guarulhos, and Joinville, although with a smaller volume of vacancies, presented a more heterogeneous distribution among the work modalities. This flexibility or less homogeneous stage of development in the data ecosystem suggests that, in these markets, there is no consolidated dominant pattern. The diversity of work formats in these centers may indicate greater adaptability of companies or a transition phase in hiring practices.

The overall frequency of skills revealed that SQL was the most recurrent, present in approximately 47% of the vacancies, consolidating itself as a fundamental transversal technical skill for data extraction, manipulation, and querying. Python appeared in about 42% of the vacancies, reinforcing its role as a basis for automation, modeling, and analysis. Knowledge in cloud computing emerged in approximately 40% of the vacancies, highlighting the growing relevance of scalable infrastructures and distributed environments.

Communication competence also showed a high incidence, being mentioned in about 38% of the vacancies. This highlights that, even in predominantly technical roles, there is a strong appreciation for the ability to translate analyses into understandable insights and to interact with business areas. Competencies such as organization (25%), AWS (22%), and Power BI (21%) indicated the importance of skills related to work structuring and the practical application of tools in the corporate context.

At intermediate frequency levels, R, Azure, Git, and Excel stood out, situated in the range of approximately 15% to 20%. Although relevant, these skills are not universally required, varying according to the job profile. More specific technologies, such as Databricks, GCP, and Airflow, showed lower relative incidence, around 10% or less, indicating that the market tends to prioritize broadly applicable and transferable competencies before demanding more advanced technological specializations.

In general, the results showed a pattern where the data market simultaneously values consolidated technical fundamentals, such as SQL and Python, modern infrastructure, such as cloud computing, and behavioral competencies, such as communication and organization. This combination reflects the interdisciplinary and applied nature of the analyzed functions, as discussed by Cimatti (2016) and Fayyad and Hamutcu (2020), who emphasize the integration of hard and soft skills.

In the Data Scientist sample, technical skills took center stage, with Python present in approximately 52% of vacancies, consolidating itself as the main tool for analysis, modeling, and development of data-driven solutions. SQL appeared in about 47%, reinforcing its importance in extracting and manipulating data from different sources. Knowledge of cloud computing emerged in approximately 38%, highlighting the need to operate in scalable and distributed environments.

In the field of behavioral competencies for Data Scientist, communication showed a high incidence, close to 37%, indicating the professional’s relevant role in translating complex analyses into understandable insights for the business. Organization appeared in about 27%, suggesting the importance of structuring analytical projects, managing experiments, and process reproducibility, according to Dubey and Gunasekaran (2015)’s view on the need for analytical and strategic skills.

At intermediate levels for Data Science, AWS and R stood out with 21%, in addition to Power BI with 19% and Azure with 18%, indicating the diversity of environments and technologies used. Skills such as leadership, Excel, Git, and autonomy were in the range of approximately 14% to 17%, reflecting complementary competencies that, although relevant, are not universally required. Relationship (10%) and teamwork (7%) showed lower relative incidence, which does not necessarily indicate low importance, but rather less explicitness in job postings.

For Data Engineer, the skills were heavily concentrated on infrastructure and data engineering, with emphasis on cloud computing, present in approximately 68% of the vacancies, configuring itself as the main technical requirement. This result shows that the role of the data engineer is deeply associated with the construction and operation of scalable and distributed environments, as pointed out by Vicario and Coleman (2019) regarding the importance of strategic data management.

SQL (53%) and Python (52%) appeared with very close frequencies for Data Engineers, reinforcing their fundamental role in the manipulation, transformation, and processing of large-scale data. AWS (40%) stood out as the main specific cloud platform, followed by Azure (31%) and GCP (20%), indicating the relevance of cloud provider domain in the development of modern data architectures. These skills are crucial for building effective data pipelines and architectures (Zaharia et al., 2016).

Among the engineering tools and practices, the relevant presence of Git (35%), CI/CD (30%), Spark (26%), and ETL (25%) was observed, highlighting the importance of versioning, pipeline automation, and distributed processing. Technologies such as Databricks, Airflow, Data Lake, and ELT, situated in the range of approximately 15% to 20%, reinforced the Data Engineer’s focus on orchestration, storage, and data structuring in complex environments.

In the field of behavioral competencies for Data Engineers, communication appeared in about 32% of job openings. Although the focus is predominantly technical, this incidence demonstrates the need for interaction with other areas, such as data science and analytics, especially in contexts of system integration and solution delivery. The results indicate that the market for data engineers is strongly driven by technical competencies related to infrastructure, cloud, and pipeline engineering, with less relative emphasis on behavioral skills when compared to the Data Scientist profile, reflecting the more operational and architectural nature of the role.

In the Business Intelligence sample, behavioral and business interface skills took center stage, with communication highlighted, present in approximately 45% of vacancies. This was the main skill demanded, reinforcing the role of the BI professional as a link between data and decision-making, responsible for translating analytical information into understandable insights for business areas, as highlighted by Davenport and Harris (2017).

SQL (41%) appeared as the main technical skill for Business Intelligence, highlighting the need for data manipulation and querying for the construction of analyses and reports. Complementarily, visualization and descriptive analysis tools, such as Power BI (33%) and Excel (31%), showed a high incidence, confirming the centrality of these technologies in the development of dashboards and monitoring of indicators, according to the literature on the importance of data visualization (Wickham and Grolemund, 2017).

Still in the field of organizational competencies, organization (30%) emerged as one of the most relevant skills for Business Intelligence, reflecting the importance of structuring metrics, data governance, and consistency in report production. At intermediate levels, Python (24%) and R (21%) stood out, indicating that, although not the main focus of the profile, more advanced analytical skills are beginning to gain ground within the BI context.

Infrastructure and engineering-related competencies, such as cloud (17%) and ETL (7%), showed a lower relative incidence for Business Intelligence when compared to Data Scientist and Data Engineer profiles, suggesting that these skills are less central to the role. In contrast, skills such as leadership (18%), autonomy (16%), relationship building (15%), and teamwork (9%) reinforced the collaborative and business-oriented nature of the function, aligning with Cimatti’s (2016) vision of soft skills.

In general, the results indicate that the job market for Business Intelligence professionals is strongly driven by communication, organization, and visualization tool proficiency, with less relative emphasis on technical infrastructure. This reflects the nature of the role as direct support for decision-making and business value generation, requiring a strong interface and data translation capability for non-technical stakeholders.

The distribution of seniority by profile revealed that, in Data Engineer, there was a strong concentration in the higher levels, with a predominance of senior and leadership positions, which together represented the largest share of vacancies. This pattern indicates that the role is strongly associated with activities of greater technical complexity and responsibility, such as the construction and management of data architectures, requiring more experienced professionals.

For Data Scientist, the seniority distribution was more balanced, although still with relevant weight in intermediate and advanced levels. A significant participation of junior and mid-level positions was noted, accompanied by an expressive presence of senior and leadership levels. This suggests a market that combines both development opportunities and demand for more consolidated professionals, reflecting the diversity of entry paths into the field.

In Business Intelligence, an even more pronounced concentration was observed in leadership positions, which represented the largest share of vacancies in this group, followed by senior and mid-level positions. This behavior suggests that the BI role is frequently associated with strategic functions within organizations, with strong involvement in indicator management, decision support, and interface with business areas, which demands experience and strategic vision.

In general, the results showed that the three areas present distinct seniority structures. While Data Engineer and Business Intelligence are more oriented towards experienced and strategic profiles, Data Scientist presents a more balanced distribution, reflecting different maturity levels and entry paths within the data ecosystem. This differentiation is crucial for understanding market expectations in each specialty.

The analysis of the quantity of skills per term, using boxplots, revealed that Data Engineer positions presented the highest median and the highest variability in the quantity of required skills. This indicates that this profile demands a broader and more diversified set of technical competencies, often associated with the construction of pipelines, data architectures, and distributed processing environments, which reinforces the inherent complexity of the role.

In contrast, Data Scientist positions showed a significantly lower median, concentrating on a more restricted number of skills, although with the presence of several extreme values. This suggests the existence of positions requiring a high degree of technical specialization, coexisting with positions of a more focused profile. Business Intelligence positions, on the other hand, showed a more concentrated distribution, with less dispersion and an intermediate median, indicating a more standardized skills profile, strongly oriented towards the use of specific data analysis and visualization tools.

The distribution of the quantity of skills by seniority, also by boxplots, showed that the junior, intermediate, and senior levels presented very close medians, around 4 skills per vacancy. This indicates a relative stability in the typical number of competencies required among these levels. However, the dispersion increased as seniority evolved, especially at the intermediate and senior levels, with wider interquartile ranges and a more significant presence of extreme values.

This behavior suggests that, although the average number of skills does not increase expressively with seniority, the variability of requirements grows, reflecting greater diversity of roles and responsibilities at higher levels. In the case of leadership positions, a lower median was observed, close to 3 skills, accompanied by a more concentrated distribution. This indicates that positions at this level tend to require a leaner set of explicitly described competencies, possibly because there is a greater emphasis on strategic, management, and decision-making skills, which are not always detailed in the same way as technical competencies in job postings.

Analysis by companies evidenced the concentration of vacancies in organizations of larger size and digital presence. Mercado Livre (68 vacancies), Itaú Unibanco (58), and Shopee (53) stood out, followed by Life (52) and Tata Consultancy Services (48). This higher volume suggests greater hiring intensity and data maturity in these companies, which is directly reflected in the complexity and specificity of the demands for skills.

The distribution of seniority by company showed a clear predominance of more experienced profiles. Companies like BairesDev presented 97.6% of vacancies in leadership levels, while Tata Consultancy Services (85.7%) and Vivo (Telefônica Brasil) (73.7%) followed the same trend. On the other hand, organizations like EY presented a greater balance, with 63.0% at the full level and 25.9% senior. This pattern indicates that, regardless of the volume of vacancies, the analyzed market tends to demand professionals with more experience, which connects with the complexity of the stacks identified.

Company specialization was also observed in the distribution of terms. BairesDev concentrated 60.0% of vacancies in Data Engineer, while EY had a predominance in Data Scientist (55.3%). Vivo (Telefônica Brasil) showed a greater focus on Business Intelligence (55.2%). Companies like Foursys (61.8% Data Engineer) and Bradesco (45.5% BI) reinforced this segmentation, evidencing that the volume of vacancies is directly associated with the type of specialization demanded by each organization.

The boxplot of the number of skills per company complemented this analysis, highlighting the technical complexity of the job openings. Companies like Vivo (Telefônica Brasil) presented the highest average number of skills (7.07) and also the highest variability (maximum of 21), indicating more demanding and heterogeneous positions. In contrast, BairesDev presented a lower average (2.73), suggesting a greater focus or specialization. Companies like EY (average 6.94) and Tata Consultancy Services (6.25) showed an intermediate pattern, reinforcing the relationship between high seniority and a greater number of required competencies.

The heatmap of average skills per company per term synthesized these relationships, highlighting clear differences between profiles. Vivo (Telefônica Brasil) stood out in Data Engineer with an average of 10.25 skills, far above the others, while EY presented a higher average in Data Scientist (7.73). In the field of Business Intelligence, companies such as Foursys (6.60) and Vivo (Telefônica Brasil) (6.44) excelled. These results reinforce that not only the number of vacancies, but also the seniority level and the type of term are directly associated with the complexity and depth of the required competencies, evidencing the existence of different hiring strategies and analytical maturity among the analyzed companies.

The SVD1 vs SVD2 map (skills x terms) presented the two-dimensional projection obtained from the application of Singular Value Decomposition (SVD) on the association matrix between professional skills and terms. It was observed that Data Engineer occupies a more isolated position at the positive extreme of SVD1 and negative of SVD2, suggesting a relatively distinct set of competencies. Skills such as SQL, Python, AWS, Azure, Spark, Airflow, and Databricks appeared close to this profile, indicating a strong association with technologies related to infrastructure, distributed processing, cloud, and pipeline engineering. This positioning reinforces the more technical and architectural nature of the term, with an emphasis on competencies focused on supporting and operationalizing the data ecosystem.

Business Intelligence presented a more isolated position in the upper quadrant, mainly in relation to the SVD2 axis. Although competencies such as Excel and Power BI were in the same relative direction of this dimension, the observed distance indicates that the Business Intelligence profile is not explained by proximity to a single competency, but by a specific combination of skills. This positioning reinforces its differentiation from other profiles, especially due to the greater relative presence of visualization tools, descriptive analysis, and business-oriented competencies, according to the literature on Business Intelligence (Davenport and Harris, 2017).

Data Scientist occupied an intermediate position between the two poles, although closer to the technical axis than to the visualization axis. Their location suggests a hybrid profile, which shares competencies with data engineering, such as Python and SQL, but without the same emphasis on infrastructural technologies. At the same time, it did not approach the more typical BI tools as closely, reinforcing its intermediate position between analytical modeling, programming, and practical application of data, according to Fayyad and Hamutcu’s (2020) description of the versatility of the data scientist.

In general, the map empirically confirmed the differentiation between the three terms analyzed. Although there is a common core of shared competencies, the spatial distribution of skills and terms showed that Data Engineer is more strongly associated with infrastructure and engineering, Business Intelligence with visualization and business support, and Data Scientist with an intermediate position, combining analytical and technical competencies within the data ecosystem. This functional segmentation is a central finding of the study.

Association graphs between skills (MCC/Phi) revealed the formation of a strongly interrelated technical block, mainly composed of competencies related to infrastructure, cloud, and data engineering. Among the most intense associations, Cloud-AWS (0.660), Cloud-Azure (0.583), AWS–GCP (0.483), Azure–GCP (0.480), and Cloud–GCP (0.470) stood out, indicating that these technologies frequently appear together, reflecting a job pattern more oriented towards building pipelines, integration, and operation of scalable environments.

Similarly, the association between Spark–Databricks (0.464), Spark–Airflow (0.418), ELT–Data Lake (0.451), and CI/CD–Docker (0.403) reinforces the existence of a core linked to modern data engineering, in which distributed processing, orchestration, automation, and storage appear as complementarily demanded skills. These associations highlight the complexity and interdependence of tools and platforms in the data environment, according to the literature on big data (Zaharia et al., 2016).

Relevant associations were also observed between more transversal skills. The SQL-Python pair (0.418) suggests that these two skills function as a joint technical base in a large part of the vacancies, articulating data manipulation and programming. Similarly, Power BI-Tableau (0.418) indicates proximity between visualization and applied analysis tools, although this block appears less connected to the more infrastructural core. In contrast, skills such as Communication, Organization, Leadership, and Autonomy showed more diffuse associations and, in general, lower intensity, suggesting that, although they appear frequently in vacancies, they are not rigidly linked to specific technical combinations.

In general, the association matrix showed that the labor market in data does not demand skills in isolation, but rather in recurrent combinations, organized into relatively coherent subsets. The main one is the engineering/cloud cluster, while a second grouping involves analysis and visualization tools. This result reinforces the interpretation that professional training in data tends to be structured in ecosystems of skills, in which certain skills gain value precisely for their ability to operate in conjunction with other technologies and practices.

The log-odds graphs, which measure the relative specificity of a skill for a given profile, highlighted in Data Engineer competencies strongly associated with infrastructure engineering and the operation of robust technical environments, such as Java, Kubernetes, Kafka, .NET, Pig, Docker, Scala, CI/CD, dbt, C#, and Git. This set highlights a profile oriented towards building and sustaining pipelines, system integration, deploy automation, distributed processing, and data architecture. Although SQL, Python, and cloud are relevant, they do not appear among the most specific as they are widely demanded in other terms, which reduces their discriminatory capacity.

In the case of Data Scientist, the most specific skills focused on statistical, analytical, and methodological competencies, notably MATLAB, Probability, Matplotlib, Bayes, Stata, SPSS, Plotly, Inference, Regression, SAS, Metabase, and Business Objects. This result reinforces the identity of Data Science as a field more strongly associated with analytical modeling, statistical inference, and the exploration of quantitative methods. The presence of analytical libraries and tools suggests that the differential of this profile lies less in infrastructure and more in the ability to build, interpret, and communicate models and analyses, according to the vision of Smith et al. (2018).

In Business Intelligence, the highest log-odds values were associated with skills related to visualization, descriptive analysis, and business interface, with emphasis on Excel, Power BI, QuickSight, Relationship, Data Studio, Business Objects, D3, Looker, Qlik, Presentation, Teamwork, and Communication. This set shows that the BI profile is marked by the ability to transform data into reports, dashboards, and useful indicators for decision-making, combining visualization tools with interpersonal and organizational skills. The presence of skills such as communication, relationship, presentation, and teamwork among the most specific reinforces that the BI professional acts more intensely in mediating between data and business, differing from other profiles by their orientation towards the interpretation, dissemination, and managerial use of information.

In summary, the results of this research revealed that the data job market in Brazil, although sharing a common core of skills such as SQL, Python, and communication, presents a clear functional segmentation between the profiles of Data Engineer, Data Scientist, and Business Intelligence. Each profile possesses a set of specific technical and behavioral skills, distinct seniority levels, and demand patterns that reflect their particular roles in the data ecosystem, providing valuable subsidies for professional training and the alignment of recruitment practices.

4. Conclusion

The present study aimed to identify and analyze the most demanded technical and behavioral skills in data-related job openings, seeking to map patterns and differences between occupational profiles and seniority levels, according to announcements published on the LinkedIn platform. It was found that SQL (47%), Python (42%), cloud computing (40%), and communication (38%) were the most recurrent skills overall. A clear functional segmentation was observed between the profiles of Data Engineer, Data Scientist, and Business Intelligence. The Data Engineer focused on data infrastructure and processing, with high demand for cloud computing, SQL, and Python, and a predominance of senior and leadership positions. The Data Scientist presented a more balanced profile, focusing on analytical, statistical, and computational skills, such as Python and SQL. The Business Intelligence professional, on the other hand, stood out for the demand for communication, visualization, and descriptive analysis skills, with a strong presence of tools like Power BI and Excel, and a concentration in leadership positions. The results indicated that skills are valued in interconnected ecosystems, and the complexity of requirements varies according to profile and seniority.

The main contribution of this study lies in offering concrete subsidies for professional training and the alignment of talent recruitment and development practices with the effective demands of the data market in Brazil, by detailing the specificities of each profile. However, the study has limitations inherent to the use of a single recruitment platform and possible biases associated with the analyzed sample. For future research, it is recommended to expand the database to include other platforms and conduct longitudinal analyses, allowing for the tracking of competency demands over time and a deeper understanding of the transformations of work in the data area.

Bibliographic References

ANTUNES, R. Os sentidos do trabalho: ensaio sobre a afirmação e a negação do trabalho. 2. ed. São Paulo: Boitempo, 2009.

CASTEL, R. As metamorfoses da questão social: uma crônica do salário. Petrópolis: Vozes, 1998.

DAVENPORT, T. H.; PATIL, D. J. Data scientist: the sexiest job of the 21st century. Harvard Business Review, v. 90, n. 10, p. 70–76, 2012.

FAYYAD, U.; HAMUTCU, H. Toward foundations for data science and analytics: a knowledge framework for professional standards. Harvard Data Science Review, v. 2, n. 2, 2020.

MARX, K. O capital: crítica da economia política. Livro I. São Paulo: Boitempo, 2013.

MENEGASSO

POCHMANN, M. Nova classe média? O trabalho na base da pirâmide social brasileira. São Paulo: Boitempo, 2012.

VICARIO, E.; COLEMAN, C. Big data and analytics: driving business performance. 2019.

WING, J. M. The data life cycle. Harvard Data Science Review, v. 3, n. 1, p. 1–10, 2021.

Article originating from the Final Course Work of the Specialization in Data Science and Analytics of the MBA USP/Esalq

To learn more about the course, click here and access the MBX Academy platform