GIL, Natália de Lacerda. Quantifying quality.md

Summary

• Federal University of Rio Grande do Sul, Porto Alegre, RS, Brazil.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

In this article, I present an analysis of the relationships, observable in the Brazilian case, between school failure rates and the measurement of teaching quality. The option was to focus the discussions on the possibility of quantifying the quality of teaching since the 20th century. Initially, it was necessary to consider the polysemy of the term quality and its implications for quantification processes. Examination of the documentation allowed us to see that, in Brazil, the naturalization of the poor school performance of students and criticism of the high failure rates, repudiated as a mechanism of selectivity and exclusion , coexisted for a long time . The link between school failure and teaching quality is, even today, perceptible in the Brazilian educational context, as observed by the analysis of the construction method of the Basic Education Development Index (IDEB) which, since 2007, seeks to quantify the quality of the school in Brazil. In this index, measuring the school knowledge that students master is not a sufficient condition to consider quality education. It has also been assumed that it is necessary to guarantee a normalized school flow, with a low incidence of failure and dropout.

Keywords: education, statistics, IDEB, education quality, teaching evaluation.

184 Natália de Lacerda Gil

http://doi.org/10.1590/15174522-109753

DOSSIER 184

Quantifying quality: some

considerations about

school failure rates in Brazil

Natalia de Lacerda Gil*

Machine Translated by Google Brazilian.1 The media constantly publicize

Quantifying quality... 185

2 As an example, see “Brazil is the second country with the worst level of learning, according to an OECD study”, published on 02/10/2016 and available at https://educacao.uol.com.br/

4 In Brazil, Basic Education is divided into three levels: Early Childhood Education, for children aged 0 to 5; Elementary Education, lasting 9 years; and High School, lasting 3 years.

3 Data available at https://agenciadenoticias.ibge.gov.br/agencia-sala-de-imprensa/2013-

1 I would like to thank Célia Caregnato for carefully reading the preliminary version and for her valuable suggestions. The problems and incompleteness of this study, obviously, are my sole responsibility.

noticias/agencia-estado/2016/02/10/brazil-and-second-country-with-the-worst-level-oflearning points to-ocde-study.htm and “São Paulo has only one school in the top 10 of the Enem 2019”, published on 06/27/2020 and available at https://noticias.uol.com.br/ultimas-noticias/

agencia-de-noticias/releases/21253-pnad-continues-2017-number-of-youths-who-neitherstudy-nor-work-or-qualify-grows-5-9-in-one-year. Accessed on 07/11/2020.

agencia-estado/2020/06/27/sp-so-tem-uma-escola-no-top-10-do-enem-de-2019.htm.

Introduction

indices and rankings that seek to highlight which would be the good schools and rejoice or are amazed, too, in reporting what basic education students know (more often, what they do not know).2

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Currently, there is a clear concern about the quality of education

Academic research and the action of private non-profit organizations have also devoted attention to measuring and/or debating the quality

of teaching. If measured by the right to access school institutions, the quality of education would have improved, but still presenting problems. It is positive that 99.2% of young people aged 6 to 14 attended school in 2017,3 but it is worth remembering that this means that more than 211,000 people in this age group were out of school (IBGE, 2018). In 2017, the average number of years of study for people aged 25 or over in Brazil was 9.1 years, with 10.1 among white people and 8.2 for the black population. If one prefers to consider school flow indices, that is, statistics that indicate whether those who enter the system regularly follow the school path established by law, we have, in general terms, the following table: between the 1st and 5th year of Education Fundamental4 failure, in 2017, was 5.2%; between the 6th and 9th years, it increased to 10.1%; in high school, the school failure rate

Machine Translated by Google 7According to the interpretation of QEdu, from the Lemann Foundation. Available at https://qedu.org.br/ gov.br/prova-brasil.

brasil/proficiencia?gclid=EAIaIQobChMIovKGq7rF6gIVVQmRCh1BygDbEAAYASAAEgLfCvD_BwE.

5 Data available at http://portal.inep.gov.br/artigo/-/asset_publisher/B4AQV9zFY7Bv/

In the socioeconomic questionnaire, students provide information about contextual factors that may be associated with performance”. Available at http://portal.mec.

content/inep-discloses-rates-of-school-performance-numbers-show-historical-trends of improvement/ 21206.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

186 Natália de Lacerda Gil

6 “Prova Brasil and the National Basic Education Assessment System (Saeb) are large-scale diagnostic assessments developed by the Anísio Teixeira National Institute of Educational Studies and Research (Inep/MEC). They aim to assess the quality of teaching offered by the Brazilian educational system based on standardized tests and socioeconomic questionnaires. In the tests applied in the fourth and eighth grades (fifth and ninth years) of elementary school, students respond to items (questions) in Portuguese, with a focus on reading, and mathematics, with a focus on problem solving.

it reached 10.8%.5 As failure is significant in Brazilian schools, we have another index that we can use: the age-grade distortion. Thus, we know that 95.5% of children aged 6 to 10 attended the initial years of Elementary School at the expected age, with no significant difference between the sexes. In the age group from 11 to 14 years old, the rate was 83.3% of men and 88.0% of women at the age at which they ideally should attend the final years of Elementary School. For those who are more concerned with knowing how much students learn, we have, for example, data from the Prova Brasil.6 In this case, a possible interpretation, based on the scores obtained in the standardized tests, is that, among 5th grade students , 56% present expected learning or beyond expectations in reading and 44% in Mathematics; among those in the 9th grade, these percentages would be 34% and 15% respectively.7 Despite the profusion of numbers, it is not unreasonable to ask: is it possible to effectively quantify quality? What is the limit of statistics in the task of allowing us to know the results of a country's education? And, after all, what is quality? The issue is thorny, as we will see later, and those who produce and analyze educational statistics generally know this. However, such numbers are formulated and presented in such a way as to make one believe that it is indeed possible to quantify quality. The perspective is encouraging and, almost always, the excitement is enough to end the discussion and move on to analyzing the numbers. Hence, the difficulty in quantifying quality is considered a minor problem, at most

Machine Translated by Google ritually at the mention of recognizing this methodological problem and follows

the enjoyment of the many advantages that quantitative analysis appears to have.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Quantifying quality... 187

the social order partly owes its permanence to the imposition of classification schemes which, by adjusting to objective classifications, end up producing a form of recognition of this order that implies precisely the lack of knowledge of the arbitrariness of its foundations: the correspondence between objective divisions and the classificatory schemes, between objective structures and mental structures, is at the root of a type of original adherence to the established order.

Reification is just one of the difficulties that must be dealt with if one intends to think with statistics. Such difficulties do not allow us to advocate the abandonment of quantitative analyses. They are useful, very useful, they undoubtedly contribute to the task of providing intelligibility to complex situations, such as education. Perhaps the problem is succumbing to the seduction of numbers and the necessary effort should be to distrust the feeling

of objectivity they give us. No, statistics are not objective. They are a useful objectification resource – which is different – and they carry risks, side effects, which we cannot ignore or disregard.

In the last three decades, an important critical literature on quantification has emphasized that statistics are not a simple description of reality, they are, on the contrary, the result of objectification processes defined by the categories used, by the choice of what to measure, by the modes to observe (Hacking, 1990; Rose, 1991, 1999; Porter, 1995; Desrosières, 2000; Popkewitz, 2011, among others). Such processes give numbers symbolic power and, being produced and used by and for people, they also produce devices of subjectivation. Bourdieu (1998, p. 117-118) highlights that

From scientific research to wide dissemination, the problem deteriorates. Because, when it makes headlines in the newspapers, the “quality” of education has already become a thing, we then have the reification of quality, and there is no need to discuss what we are actually talking about again. The numbers gain fluidity and circulate happily, from news to news, allowing for good insights, but also almost any type of interpretative abuse.

Machine Translated by Google It is in the sense of contributing to a critical analysis of education statistics that, in this article, the focus is on the ways of quantifying the quality of teaching. More especially, what I intended was to scrutinize how, in the Brazilian case, school failure was taken as an indicator of the lack of quality in Brazilian schools.8 To do so, I return the analysis to the beginning of the 20th century, when we have for the first time the systematic organization of education statistics in the country. This retreat is explained by the intention to know how the arguments were presented when establishing the democratic school in Brazil.9 As suggested by Bourdieu (2014, p. 103), “the interest of the return to the genesis is that it is very important because there is debates in the early days in which things are said in all letters that later appear as provocative revelations to sociologists”. The text is organized into three parts. Firstly, using the available literature on the subject, although without being exhaustive, I analyze the notion of quality in education. The polysemy that characterizes the term makes the task difficult, but, even so, such an examination is unavoidable if the intention is to quantify quality. Next, I locate, in Brazilian educational debates, traces of what were considered to be elements articulated around school quality and its measurement. In this sense, it is important to observe that the quantification referred, initially, to the characteristics and abilities of the student body; over time, it also began to be mobilized to denounce the school's inability to comply with

Statistics produce what Bourdieu (1998) called a “ theory effect”, that is, as they seek to provide forms of intelligibility of the world, they collaborate in the construction of the conditions of existence of what they intend to describe.

188 Natália de Lacerda Gil

Such defenses had a strong influence on the way in which the Brazilian school was established in legislation.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

8 In common sense, however, a high failure rate is often considered an expression of quality. Even many professors still think this way, despite official documents and specialized literature for more than a century indicating the opposite. However, I will not deal with this in this article, although it is a very important issue and still little studied.

9 In the 1930s, liberal educators defended the expansion of the democratic school, understanding it as an institution that would guarantee equal opportunities for all and would proceed with a selection that was considered fair because it was based on the innate talents of students.

Machine Translated by Google Several authors highlight that the concern with the quality of education does not cancel out the continuous attention that must be given to guaranteeing access to schooling, embodied in the existence of places in schools in sufficient quantities to meet all demand. In this sense, it is worth noting that, as education has been proposed in most

After all, what is quality? This is difficult terrain, full of rocks and tripping hazards. Initially, it is important to state that quality, especially in education, is a polysemic concept. As for this observation, the many authors who dedicate themselves to the examination of the question are in agreement. There are several elements to take into account. Without intending to establish a temporal sequence and, even less, any hierarchy, let's start by noting that it is a historical concept, because it changes over time, articulating itself to values, knowledge and expectations socially shared in different periods. It is frequent that the mention suggests a linear and progressive temporal succession, which leads to believe that the variation of meaning of the term quality of education would establish, in the historical transitions, total alterations of meaning. Now, as we intend to argue throughout this article, the analysis of historical processes leads us not to the understanding of a sequence of meanings of quality that would replace their precedents, but to the perception of the coexistence of meanings formulated in different temporalities. Another aspect to be observed refers to the understanding, in the case of education, that quality and quantity are complementary.

its social role. Finally, I present the concepts that guide the construction of the Basic Education Development Index (IDEB), created in 2007, and present some of its limits and recurrent criticisms. The intention was not to carry out a detailed historical analysis, but rather to pursue, with methodological rigor, the arguments that coexist, often in a contradictory way, in debates on the quality of Brazilian education over time.

How to quantify quality in education?

Quantifying quality... 189

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google In contemporary societies as a universal right, quality education only for a few people configures privilege and, therefore, lack of quality from a social point of view.

190 Natália de Lacerda Gil

Understanding quality and quantity as articulated aspects makes it possible to identify why, in Brazilian history, as in other countries, the discussion appears to be strongly associated with enrollment totals. But it doesn't hurt to remember a fact already widely evidenced in several studies: “extending schooling is not, by itself, a guarantee of school justice. In fact, everything depends on how the school is organized and what is done in it” (Dubet, Duru-Bellat; Vérétout, 2012, p. 35). The notion of quality in education associated with guaranteeing access for all to the same school – of quality – has some pitfalls. Defending that everyone has access to a quality school seems a noble and indisputable intention in democratic societies. However, in this case, who is it about quality?

In what perspective? In line with which society project? As Esteban (2008, p. 6-7) highlights,

quality is a polysemic, plastic word, which contains virtualities and positivities, expresses a convergence of concerns, allowing the rapid construction of a consensus by creating the idea of aggregation around common commitments. These characteristics hide how much their different meanings hold opposing and contradictory possibilities of organizing the school as a social project.

By obliterating the polysemy of the term, the meaning constituted in specific contexts is assumed to be universal and timeless. This proceeds to “maintain the historical process of coloniality of power in which relations of subordination are woven” (Esteban, 2008, p. 7). Regarding the discourses that refer to the rights of “all children”, Popkewitz (2011) observes that they give rise to practices of identifying those who do not share the characteristics of “all children”, that is, those who would need to be rescued and saved from their original contexts because they were considered inadequate. When a standard is established

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google On the other hand, the function of the school, in the context of globalization and in the face of neoliberal policies, strengthens the focus on preparing individuals with mastery of applicable knowledge and in compliance with demands of an economic nature (Stromquist, 2012). Thus, the standardization of school content and evaluation processes is defended as decisive for guaranteeing the quality of teaching. In this sense, it is necessary to remember that, in recent decades, the discussion about quality has been particularly prominent in business circles and this is not without consequences in the area of education. According to Oliveira and Araújo (2005, p. 7),

On the one hand, there is the understanding of the role of the school for human development, on the other hand, there are arguments that advocate the role of the school for economic development. In the first case, the school would be associated with the construction of a democratic society. Virginio (2012, p. 179) highlights the difficulty of guaranteeing the full performance of this function in the face of standardized evaluation criteria:

Thus, perhaps the most crucial aspect for the discussion on polysemy is the notion that quality is linked to the conception of education considered. That is, different educational paradigms – or, if you prefer, educational projects – will engender distinct quality criteria and different perceptions of what quality education is. If it is a fact that there are many conceptions of education that can be described, it happens that two are the most recurrent in the debates around the theme.

unique quality, there is the production of invisibility and subalternity of subjects who do not fit the standard (Esteban, 2008).

This distinction is important as it draws attention to the fact that the concept of quality, even in the business world, carries significant meanings.

Quantifying quality... 191

public policies and the efforts that have been undertaken, in the sense of guaranteeing school success, can ensure a better performance in terms of expectations of success in relation to certain curricular prescriptions. However, they are insufficient to account for the need for an education consistent with a democratic society. It is about thinking beyond the criterion of merit, or even the instrumental character of knowledge.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google and different procedures. One should also not lose sight of the fact that a significant part of the debate on quality in education is imported from the business world and, even so, in this restricted scope, it embodies different meanings.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

192 Natalia de Lacerda Gil

Thus, statistics cannot be a basic element to assess what is important – as if only what is expressed in numbers were important.

If there is no agreement on what quality of education is, how would it be possible to quantify it? The answer to this question is political, but it also involves technical aspects. It is a political issue because it depends on which social group has the most strength to impose its way of seeing. The elites have greater power to reify quality, according to their conceptions of school and society (Bourdieu, 2014). Having become a “thing”, quality can be quantified, inevitably leaving out of the count a wide range of aspects whose importance in the educational process is undeniable (such as the development of empathy and learning to respect others, the establishment of links between different generations, building self-esteem, creative ability, among others). Now, this is an unavoidable effect of quantification. As Afonso (2009, p. 13) argues , “not everything that counts in education is measurable or comparable”.

The answer to the possibility of quantifying quality is also technical, because it implies excluding everything that, even if consensually considered relevant, cannot be expressed in numbers: for example, how to objectively assess students' satisfaction in learning new knowledge? How to know the development of students regarding

According to these authors, in some circumstances, the quality of education has referred to the “final product”, the student who learned, or it may also be an indication of a low cost process, especially low cost for the State.

This becomes a problem when one loses sight of the fact that the statistical representation of a phenomenon is just a way of seeing it, meeting the criteria of the spaces of power; it may even be very useful in certain circumstances, but it does not correspond to the totality of what is accounted for.

Machine Translated by Google Quality of students or quality of teaching?

Quantifying quality... 193

self-esteem, social skills and creativity? Or how to measure the importance of the bond between generations established throughout the educational process? The risk is that, as they cannot be expressed in objective indices, attention and time will not be devoted to such aspects that are essential in youth development processes. This shows that the choice of categories in statistical surveys, supposedly a purely technical issue, has an indelible political dimension. As Rose (1991, p. 674) states,

The lack of consensus and all the complexity involved in defining what “quality of education” means, however , have not prevented quantification from being carried out for a long time. Tacitly operating with this multiplicity of issues, concepts and actions to quantify the quality of education were overlapping over time. In the Brazilian case, as we will see below, measurement dates back to the 1930s and was improved amid the debate about what and how to quantify in quality. A brief historical excursion allows us to examine the arguments and note the origin of the “layers” of meaning that exist in the discussion today.

School failure was not always a problem in Brazilian schools (Gil, 2018). Although since the installation of the first colleges, still in the colonial period, there were school exams that students could fail – a practice that continued in public classes during the imperial period –, this occurrence is not mentioned in the documentation of both periods as frequent or as problem. The apprehension of school failure as a problem stems basically from the association of

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

paradoxically, in the same process in which numbers reach a privileged status in political decisions, they simultaneously promise a “depoliticization” of politics, redrawing the boundaries between politics and objectivity, intending to act as automatic technical mechanisms to make judgments, prioritize problems and allocate the scarce ones. resources.

Machine Translated by Google 11 10 For a deeper understanding of the conceptions and criticisms of IQ tests, see Gould (2003).

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

194 Natalia de Lacerda Gil

However, even before this school phenomenon gained prominence and could be statistically measured, the quantification of student performance took its first steps in Brazil. Following the example of France, initiatives were organized to apply psychological tests to students in order to identify their capabilities and classify them by intelligence level. It is important to note that the development of such measurements in France took place at the request of the Ministry of Education, which intended to solve the problem of students lagging behind in school learning, selecting them to be taught in special classes, adapted to their needs. The difficulty in ensuring the democratic principle of the right to education for all resulted in the demand made by the French government to Alfred Binet, in 1904, to “develop techniques for identifying children whose school failure suggested the need for some form of special education” (Gould , 2003, p. 152).

two aspects: on the one hand, the progressive adherence of Brazilian society to the notion of democratizing schooling as a right and, on the other, the production, since the 1930s, of more reliable, comprehensive and regular education statistics.

In Brazil, such tests were assumed with optimism by educators at the head of the school democratization project, in the 1920s and 1930s, who were fully convinced of meritocratic principles and saw in schools an institution capable of carrying out the fair selection of innate talents. 11 Statistics, in this sense, fulfilled the role of measuring the distance of each individual in relation to the standard norm. Thus, a spectrum of abnormality was created (Lima, 2018) which, based on performances in psychological tests, served to classify students and engendered processes of subjectivation. As Rose (1999) points out, statistics are part of

The intelligence tests developed by Binet allowed the creation of scales for measuring IQ (intelligence quotient) and these scales traveled the world, being adapted to different national contexts.10

It is important to note that the understanding of innate talents as a fair resource for the selection and classification of students will have a long life in educational debates. For a critique of the ideology of natural aptitudes, see Bisseret (1974).

Machine Translated by Google “technologies of the self” collaborating in the establishment of forms of self-government, considered appropriate in a given historical moment.

in the initial classification for class groupings and application of teaching programs etc., and in the periodical evaluations of the intimate process of educational work and its practical effects, considering the extension and composition of these programs, in addition to other objective tests, the mental tests and pedagogical or schooling tests, those for the verification and measurement of mental qualities, these for the verification of the students' performance in the different disciplines.

The Brazilian educators who occupied prominent positions in the bodies responsible for conducting educational policies, in the period, showed strong adherence to liberal ideals. In the first decades of the twentieth century, they vigorously defended the democratic school, understanding by this expression a school in which absolutely all children were guaranteed enrollment, whether boys or girls, poor or rich. Enrollment guaranteed to all, the school would select, through its own processes, the best of each generation. These would be provided with a long schooling, so as not to waste their talents. For the others, it would be foreseen courses of preparation for work, leading each person to the position consistent with their abilities, so that each one would give the best of himself for the progress of the nation. As an example, observe what Teixeira de Freitas, one of the founders of the IBGE and who, for many years, was at the forefront of the statistics produced by the Ministry of Education and Health, says: “to be fair, education must be extended to all citizens, with no privileges save the natural endowments of personal receptivity.” (Freitas, 1945, p. 348). Jardim (1946, p. 460, emphasis added), from the Education and Health Statistics Service of the Ministry of Education and Health, in line with his contemporaries, considered tests to be a fundamental resource for organizing education:

Quantifying quality... 195

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Rose (1999) also points out that statistics are also ways of producing knowledge about students in order to support government actions on specific populations, by proposing strategies for reform and prevention of behaviors considered undesirable in the exercise of micropower.

Machine Translated by Google the new student appears as an unknown; always offers a field for the development of hopes: perhaps he is strong at work, an element that fits well with the group, interested in school activities, compensating for the efforts spent. The repeater is a failure – he has already revealed himself, nothing is expected of him. Is it really worth keeping him in the school, filling a vacancy and preventing, with his presence, the admission of another student?

failures are exact numerical data; but the criteria that determine them, as we know, change from school to school. Will the 60% promoted in school A be equivalent, in terms of educational level, to the 60% in school B? Perhaps yes; maybe not. It is the school authority that examines – the inspector or principal – and as there are “low” authorities, who are satisfied with little, there are also “high” ones, willing to tighten the scrutiny of approvals (São Paulo, 1936, p. 3 ).

There is also mention of the recognition that, as school approval was based on the results of the exams, it was possible for teachers to start training students specifically for the tests, as a way of improving the rates. Almeida Junior stated that “the fact that we focus attention on the phenomenon of failures does not mean

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

196 Natalia de Lacerda Gil

But there is also, in the period, a specific debate about the measurement instruments. The criteria considered by the teachers during the exams differed from one school to another. Thus, for example, to explain the approval rate of 1935, which was a little more favorable in some educational institutions, Almeida Junior, who held the position of manager of the state education system in São Paulo, mentioned the variation in exam requirements as an explanation for differences:

In this sense, it is important to note that, in the early debates, the notion of quality was associated with the innate abilities of students. In other words, the problem of low performance in Brazilian schools, expressed in high rates of failure and school dropout, would be, at least in part, due to the lack of quality of the children who arrived at school . There was an evident naturalization of low school performance in Brazil, sometimes even expressed as inevitable. Cardoso's statement (1949, p. 74, author's emphasis) serves as an example :

Machine Translated by Google Since the 1930s, Teixeira de Freitas indicated that the exclusive concern with

the expansion of the school, verified by the increase in enrollments, was insufficient.

According to him, the main problem of Brazilian schools concerned its low efficiency,

since the school had difficulty attracting the school-age population, as well as

maintaining attendance and guaranteeing approval of students from one grade to

the next. He was against the uncritical expansion of a school that, in his opinion,

needed to improve.

that the only thing we are asking of schools is to prepare students for exams” (São

Paulo, 1936, p. 3).

Anísio Teixeira (1935, p. 74), who was Director of Public Instruction in the

Federal District, also emphasized that the school failed to fulfill its social function by

failing and rejecting a significant part of the student body:

Throughout the period, the analysis of debates on Brazilian education allows

us to see the emergence of a discussion that shifts the cause of failure: from

evidence of the low capacity of students, it also becomes possible to indicate the

inability of the school to fulfill its social role . . So let's move on to this debate.

That's why I considered that

[...] the slogan that best expressed the demands of Brazilian children in terms of primary education would have to be, in our view, this: “before more schools, better schools”. Understood as such, an inviting School, which retains, protects, teaches and truly educates the children entrusted to it (Freitas, 1946, p. 43).

Quantifying quality... 197

it is not enough to have schools for the most capable, it is essential that there are

schools for everyone. It is not enough to have schools for everyone, it is essential that

everyone learn.

It is not difficult to assess how much the modification came to influence the concept

of school performance. Before, given the selective character, failure was almost the

index of teaching quality. If many failed, this meant that the judging criteria were really

efficient and the fine flower of the population was being purified for the formation of

intellectual and professional elites.

If, however, the school has the duty to teach everyone, because everyone needs the

fundamental elements of culture to live in modern society,

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google Such debates gave rise to policies that adopted alternative forms of assessment and student flow in order to reduce school exclusion

(Mainardes, 1998). From the 1960s onwards, reaching its peak in the 1990s, the discussion on school flow problems stands out, with an emphasis on reporting

the problem is reversed. A failed student does not mean the success of the selection apparatus, but the failure of the institution of fundamental preparation of citizens, men and women, for common life.

But this debate only echoed in the area of education from the 1950s onwards, when criticism of the high rates of school failure increased, sometimes associated with explanations for school failure that blamed students and their families (Patto, 1993), sometimes around the

discussion about a solution to the problem. In this sense, it is interesting to note that the Revista Brasileira de Estudos Pedagógicos, official journal of the National Institute of Educational Studies and Research Anísio Teixeira (INEP), will serve as a space for assessing the advantages and risks of establishing in Brazil the automatic promotion (Fernandes, 2000). The arguments in favor pointed to the recognition that education as a right for all could not be anchored in selective pedagogical practices. Luiz Alves de Matos, for example, in an article published in 1956, pointed out “that the elementary school is a matter of law and that it should not become an agency for selecting privileged talents, but should be a disseminator of education and culture at the service of youth and democracy'” (Fernandes, 2000, p. 82). The opposing arguments did not differ from the principle of the democratic school, but warned of the risks of hasty action that could be inappropriate for the Brazilian context. In this sense, Luiz Pereira, in an article from 1958, quoted by Fernandes (2000, p. 84), denounced

198 Natália de Lacerda Gil

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

that repetition is the consequence of a series of serious problems and that automatic promotion, although “eliminating high percentages of repetition, would not directly and profoundly affect the factors of this phenomenon and would lead to the loss of a valuable thermometer of the functioning of the system primary school – repetition rates”.

Machine Translated by Google Quality measurement: IDEB as a measuring instrument

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Quantifying quality... 199

The end of the 20th century was also the period in which there was wide diffusion of neoliberal policies. It was important, in this sense, to achieve maximum efficiency with a minimum of public investment. Although States still want schools to guarantee some level of social cohesion, in this new context, they tend to reduce their participation in the provision of public education (Stromquist, 2012). Alternatively, they began to focus attention on processes of privatization, decentralization and accountability. Among the control strategies that characterize neoliberalism, great importance has been given to large-scale evaluations, which aim to measure the quality of teaching as a way of inducing its increase, for example, through the competitiveness generated between schools.

The elimination of school failure raised, however, questions about whether it would not be just a way of “formally solving school failure (failure rates), but not the real problem – that of student learning [...]” (Mainardes, 1998 , p. 25). In other words, these policies resulted in a significant reduction in the failure rates in the networks where they were implemented, but they suffered several criticisms and generated distrust about the ability to maintain the quality of education in this way.

the marginalization and exclusion of the poorest. Some important public education networks began to adopt cycles or continuous progression in the organization of their schools, abandoning the annual serial progression that provided for the possibility of failure at the end of the year, if the student did not reach a sufficient score in the assessments carried out by each teacher.

From the 1990s onwards, guidelines on education produced by international organizations emphasized the notion of equity as an essential aspect (Klein, 2017). The World Conference on Education for All, held in 1990 in Jomtien (Thailand), reiterated education as a fundamental right for all and the World Declaration on Education for All:

Machine Translated by Google 12 INEP conceptualizes school dropout as the situation of a student who stops attending school before the end of the school year, without having formalized a transfer request. Available at http://download.inep.gov.br/educacao_basica/censo_escolar/

satisfaction of basic learning needs, signed by the participating countries , established that “basic education must be provided to all children, youth and adults. Therefore, it is necessary to universalize it and improve quality, as well as to take effective measures to reduce inequalities” (Unesco, 1990, p. 4). The intention to guarantee a “ minimum standard of learning quality” was linked to the recommendation to “implement performance evaluation systems” (Unesco, 1990, p. 4). Brazil followed this movement and, since that period, began to build instruments to measure the quality of education that intended to assess what students learned at school. In 1991, the Ministry of Education instituted the Basic Education Assessment System (SAEB) whose objective was to assess the quality of teaching through a sample assessment, carried out every two years, of student performance in standardized tests (Bonamino; Sousa, 2012) . In 1998, the National High School Examination (ENEM) was created and, in 2005, the Prova Brasil was added to the system , establishing non-sampling forms of assessment that would allow the individualization of results by school. More than providing parameters for measuring the teaching carried out, such policies aim to direct curricular contents and pedagogical work. Bonamino and Zákia Sousa (2012, p. 380, authors' highlights) analyze that

However, as, in the Brazilian case, the guarantee of access to basic school was delayed and as permanence is not assured, the concerns with the measurement of learning could not be disarticulated from the measures of failure and school dropout,12 phenomena that are still very recurrent.

the dissemination media strategy, through rankings, although not official, together with the distribution in schools of the content and skills matrix used in the preparation of Portuguese language and mathematics tests, introduces concrete perspectives of more direct interference in what schools do and how they do it.

200 Natalia de Lacerda Gil

nota_tecnica/2015/note_tecnica_indicadores_de_rendimento_2012.pdf.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google an educational system that systematically fails its students, causing a large number of them to drop out of school before completing basic education, is not desirable, even if those who complete this stage achieve high scores in standardized exams. On the other hand, a system in which students complete high school in the correct period is not of interest if they learn very little. In short, an ideal system would be one in which all children and adolescents have access to school, do not waste time with repetition, do not drop out of school early and, at the end of it all, learn.

Thus, a school whose students show good proficiency in the Prova Brasil, has its IDEB penalized if it presents high rates of failure and dropout. This evidences, in the conformation of the instrument, the concern with the fulfillment of the legal precept of education as a right of all. It is also consistent with what the National Education Plans (2001-2010 and 2014-2024) have been proposing when they indicate, among the goals, the demand for regularization of school flow, intending to progressively make failure and dropout rare events. In 2014, the PNE began to include quantitative targets, however, without specifying the basis on which the averages intended to be achieved were established. Detached from the assessment of the objective conditions for achieving the indicated goals, the numbers become random and express wishes disconnected from reality.

in the country. In 2007, therefore, INEP created an index that associated both aspects. As stated by Fernandes (2007, p. 7),

With regard to performance, the data come from the Prova Brasil applied every two years to students enrolled in the 5th and 9th years of Elementary School and in the 3rd year of High School. With regard to performance, passing allows assessing problems in the school flow, since the calculation is made by the difference between those enrolled at the beginning of the year and those approved, thus allowing to know the numbers of failure and dropout.

With that in mind, at that time, the Basic Education Development Index (IDEB) was proposed , whose formula associates information on student performance in standardized Portuguese and Mathematics exams and on performance, by pass rates.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Quantifying quality... 201

Machine Translated by Google The IDEB was proposed as a way to establish a quality standard in education in the country. This is an index designed to monitor results by school, with the aim of providing information to education network managers, supposedly accessible information to families about the institutions where their children are enrolled, as well as elements for evaluating Brazilian education as a whole. . Thus, it is intended that the index serves to guide educational policies as well as for social control over the quality of teaching. The specialized literature has, however, pointed out some of its risks and limits, many of which result from the inappropriate use of numbers. A central aspect, in this sense, is that educational indexes foster the illusion that it is possible to know objectively the quality of educational institutions. question the

5.7

4.7

IDEB

5.2 5.0 Early Years of Elementary School

5.2

Source: PNE 2014-2024 4.3 Final years of Elementary School

High school

2015 2017 2019

4.7 5.0

5.2

13 Although the IDEB has a scale from 0 to 10, it does not behave like the traditional school assessment. Depending on how the calculation is done, 6.0 corresponds to a high IDEB and the difference between 4.0 and 6.0 is greater than it might seem. Grades below 2.0 and above 8.0 are extremely rare. For more information on technical aspects, see Soares and Xavier (2013).

2021

5.5 6.013

5.5

Table - National Education Plan 2014-2024 - projected averages

Goal 7: To promote the quality of basic education in all stages and modalities, with improvement of the

school and learning flow in order to reach the following national averages for Ideb

use of the results of large-scale assessments and the indices created by them as the only source for analyzing the work carried out by schools, considering that the average performances obtained cannot be

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

202 Natalia de Lacerda Gil

Goal 7 of the National Education Plan explains that it intends to establish, for Brazil, tangible goals in terms of quality of education and make accountability viable, for which quantification is considered fundamental.

Machine Translated by Google It is also necessary to bear in mind that the IDEB does not allow the assessment of added value (that is, the difference in proficiency between the beginning and the end of schooling) and, also, that the standardized proficiency exams do not assess content that is important to the training of students, restricting up to reading and math. For example, components such as Geography, Art and Science are left out , or even competences such as valuing the diversity of knowledge, caring for oneself, others and the planet, collaborating with the construction of a just society, democratic and inclusive (Brasil, 2018, p. 9).

In addition, there has been unanimous criticism of the fact that the index disregards the students' socioeconomic level, leading to the belief that schools with a high IDEB would have better teaching than the others, when it is known that the correlation between student performance and socioeconomic level is always high.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Quantifying quality... 203

Thus, by attributing to this indicator the status of synthesis of the quality of education, it is assumed that the school can overcome all exclusion promoted by society.

translated as a faithful portrait of the quality of institutions (Almeida; Dalben; Freitas, 2013, p. 1155).

14 It should be noted that there are differences between what the technical teams producing statistics want from the numbers and what the managers intend when ordering the creation of indexes. For these, the objective of inducing policies provided by the dissemination of statistics is often more explicit, which, in their speech, appears more as a distortion or abuse in the use of numbers. This aspect was not systematically verified here, but it constitutes an important issue that deserves more accurate studies.

In spite of these considerations, widely known and considered by statisticians and researchers who study the subject, the circulation of numbers is detached from these reading keys and the figures gain autonomy. The effects of the indices go beyond their initial propositions.14

There is a wealth of literature that shows that this is impossible. All students have the right to learn, and the knowledge and skills specified for basic education must be the same for all. However, obtaining this learning in schools that serve students who bring less of their families is much more difficult, a fact that must be considered when using the learning indicator to compare schools and identify successes (Soares, 2011).

Machine Translated by Google Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

the use of an indicator as the sole measure of school and systems quality will naturally make schools seek to maximize it and, as this can be done in ways that are pedagogically inappropriate, it can lead to a dysfunctional educational system.

204 Natália de Lacerda Gil

Less than a functioning distortion, this is one of the characteristics of quantification processes.

Assuming education statistics as an object of analysis, seeking to scrutinize the way they are constructed and interpreted, corresponds here to considering them as participants in educational processes – and not mere measuring instruments. With regard to the quantification of quality, the difficulty already begins with the definition of what an education of

Thus, for example, the reception of the index by schools can result in actions whose only objective is to improve the index and not necessarily to improve the quality of the pedagogical work. Soares and Xavier (2013, p. 915), point out that

It is therefore necessary to consider the subjectivation effects that IDEB causes in the school space (Klein, 2017). Establishing a set of aspects from which quality is inferred, even if it is a methodologically relevant resource, comes up against the fact that reception of the index ends up limiting efforts only with regard to this restricted set. That is, if the verified proficiency is limited to reading and mathematics, it is not necessarily because the formulators of the index consider that this is the only task of the school, but because this would be a sufficient parameter for a standardized calculation. However, school subjects, education network managers and the mainstream press tend to receive this methodological restriction as a pedagogical prescription regarding the curricular contents with which they should deal. Such effects are beyond the control of statisticians, causing the measuring instrument to act in the construction of what it supposedly should only measure.

Final considerations

Machine Translated by Google describe the objective mechanisms that determine the continued elimination of disadvantaged children. Indeed, it seems that the sociological explanation can fully account for the differences in success that are most often attributed to differences in gifts. [...] The cultural heritage, which differs [...] according to social classes, is responsible for the initial difference of children in the face of the school experience and, consequently, for the success rates (Bourdieu, 2012, p. 41-42 ).

Quantifying quality... 205

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

quality. Far from being a simple and consensual issue, what I sought to underline was the fact that such conceptions are the result of sociohistorical processes that, in the midst of constant disputes, engender the coexistence of meanings that do not always agree. Such meanings, in turn, are mobilized in current debates, with varying weight and effectiveness. The second element that I intended to highlight in this article was the understanding, resulting from the analysis of historical sources, that, frequently, what is considered as a measure of the quality of teaching is the examination of the student's capacities. Here it is worth explaining an important sociological issue, addressed by Bourdieu in several of his works. The analysis of the quantification processes of school failure has contributed to

The tradition of assessing the quality of teaching by measuring what students know is maintained at IDEB. Although this index seeks to balance verification of student performance in standardized tests with quantitative data on school flow, the focus remains limited to students. There is no mention of the infrastructure conditions of the schools, there is no consideration given to the training and remuneration of teachers and other education workers, the public and/or private expenditure per student is not taken into account, that is, the socioeconomic level of the students is not included in the calculation. families and resources devoted to education. This way of quantifying quality suggests an equation that is too simple, leaving out of the analysis an important part of the complexity of the social structure that permeates and constitutes the school institution.

Machine Translated by Google Natalia de Lacerda Gil has a PhD in Education and is a professor and researcher at the Faculty of Education of the Federal University of Rio Grande do Sul (UFRGS). ÿ [email protected]

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

206 Natália de Lacerda Gil

Thus, one of the effects of quality quantification that I intended to emphasize in this article concerns the excessive simplification of the issue of quality in education expressed by statistics. If, on the one hand, numbers are a useful objectification resource that helps in the analysis of complex situations and in making political and educational decisions, on the other, they lead us to assume that everything that matters would be expressed statistically. As Desrosières (2000, p. 9) highlights, it is necessary to understand how social facts are transformed into statistics, taking into account that “the history of its gestation allows us to outline, retracing old controversies and debates, a space for articulations between technical languages and their uses in social debate”. From my point of view, it is not a question of abandoning the use of numbers, but of being more attentive to its risks and limits (Gil, 2019).

Since quality education is a right guaranteed by Brazilian legislation, it is necessary to take into account yet another fundamental aspect. In a society enthusiastic about “objectivity”, to tell something is, as already mentioned, to describe it, to make it exist. Hence a dilemma: how to guarantee the right to education without measures, without parameters, without statistics? Given this, some authors defend the indispensability of objectively establishing quality standards in education (Oliveira; Araújo, 2005). The question is not of little importance, but its positive aspect does not eliminate the risks already mentioned. The space for clashes – of ideas and policies – is, of course, open. Understanding statistics as an element mobilized in this game is already a less restricted way of taking it into consideration.

Machine Translated by Google References

Quantifying quality... 207

9. CARDOSO, Ofélia B. The problem of grade repetition in primary school.

10. AFONSO, Almerindo J. Not everything that counts in education is measurable or comparable. Criticism of accountability based on standardized tests and school rankings. Lusophone Journal of Education, n. 13, p. 13-29, 2009.

11. ALMEIDA, Luana C.; DALBEN, Adilson; FREITAS, Luiz Carlos de. IDEB: limits and illusions of an educational policy. Education & Society, v. 34, no. 135, p. 1153-1174, 2013.

Brazilian Journal of Pedagogical Studies, v. 13, no. 35, p. 74-88, 1949.

3. BISSERET, Noële. The ideology of natural aptitudes. In: DURAND, José Carlos. (org.). Education and class hegemony. Rio de Janeiro: Zahar, 1974. p. 30-67.

4. DESROSIÈRES, Alain. La politique des grands nombres : histoire de la raison statistic. Paris: La Découverte, 2000.

5. BONAMINO, Alicia; SOUSA, Sandra Z. Three generations of basic education assessment in Brazil: interfaces with the school curriculum. Education and Research, v. 38, no. 2, p. 373-388, Apr.-Jun. 2012.

6. DUBET, François; DURU-BELLAT, Marie; VÉRÉTOUT, Antoine. School inequalities before and after school: school organization and the influence of diplomas. Sociologies, v. 14, no. 29, p. 22-70, 2012.

7. BOURDIEU, Pierre. The economy of linguistic exchanges: what talking means. São Paulo: EDUSP, 1998.

8. ESTEBAN, Maria Teresa. Silencing polysemy and making subjects invisible: questions about the discourse on the quality of education. Portuguese Journal of Education, v. 21, no. 1, p. 5-31, 2008.

9. BOURDIEU, Pierre. The conservative school: inequalities in school and culture. In: NOGUEIRA, Maria Alice; CATANI, Afrânio (Orgs.). Education writings. 13. ed. Petrópolis: Vozes, 2012. p. 39-64.

10. FERNANDES, Claudia de O. Automatic promotion in the 50s: a bibliographic review in the RBEP. Brazilian Journal of Pedagogical Studies, v. 81, no. 197, p. 76-88, 2000.

11. BOURDIEU, Pierre. About the State: courses at the Collège de France (1989-92).

12. FERNANDES, Reynaldo. Basic Education Development Index (Ideb). Brasilia: National Institute of Educational Studies and Research Anísio Teixeira, 2007.

São Paulo: Companhia das Letras, 2014.

15. FREITAS, Mário AT de. New goals for education in Brazil. Brazilian Journal of Pedagogical Studies, v. 4, no. 12, p. 346-360, 1945.

16. BRAZIL. Ministry of Education. Common national curriculum base. Brasília: MEC, 2018.

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Machine Translated by Google 208 Natália de Lacerda Gil

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

24. LIMA, Ana Laura G. The “problem-child” in the Brazilian school. Curitiba: Appris, 2018.

25. Rio de Janeiro: IBGE, 1946.

26. GIL, Natália de L. School failure in Brazil: history of the configuration of a political-educational problem. Brazilian Journal of Education, v. 23, p. 1-23,

28. MAINARDES, J. Automatic promotion in question: arguments, implications and possibilities. Brazilian Journal of Pedagogical Studies, v. 79, no. 192, p. 16-

29. GIL, Natália de L. Statistics and education: considerations on the need for a careful look. Thinking Education in Review, v. 5, no. 2, p. 1-29, Jun.-Aug. 2019.

29, 1998.

19. GOULD, Stephen J. The Mismeasure of Man. São Paulo: Martins Fontes,

21. MATOS, Luiz A. de. School approval and failure. Brazilian Journal of Pedagogical Studies, v. 26, no. 63, p. 254-257, 1956.

22. HACKING, Ian. The taming of chance. Cambridge: CambridgeUniversity,

23. OLIVEIRA, Romualdo P. de; ARAUJO, Gilda C.de. Teaching quality: a new dimension in the fight for the right to education. Brazilian Education Magazine, n. 28, p. 5-23, 2005.

nineteen ninety.

28. PATTO, Maria Helena S. The production of school failure. Sao Paulo: TA

29. IBGE – Brazilian Institute of Geography and Statistics. Education 2017. Continuous PNAD. Rio de Janeiro: IBGE - Directorate of Research, Work and Income Coordination, 2018.

Queiroz, 1993.

22. GARDEN, Germano. Collecting educational statistics IV. Brazilian Journal of Pedagogical Studies v. 7, no. 21, p. 452-463, 1946.

23. PEREIRA, Luiz. Automatic promotion in primary school. Brazilian Journal of Pedagogical Studies, v. 30, no. 2, p. 105-107, Oct./Dec. 1958.

24. FREITAS, Mário AT de. Brazilian primary education in the 1932-

25. KLEIN, Delci H. IDEB and machinery: the production, quantification and expression of the quality of Brazilian education. 2017. Thesis (Doctorate in Education). Education University. Federal University of Rio Grande do Sul, Porto Alegre, 2017.

26. POPKEWITZ, Thomas. PISA: numbers, standardizing conduct, and the alchemy of school subjects. In: PEREYRA, Miguel A.; KOTTHOFF, Hans-Georg; COWEN, Robert (eds.). PISA under examination: changing knowledge, changing tests, and changing schools. Rotterdam: Sense Publishers, 2011. p. 31-46.

Machine Translated by Google Quantifying quality... 209

Sociologias, Porto Alegre, year 23, n. 56, Jan-Apr 2021, p. 184-209.

Failure in primary school (The phenomenon of failure. Analysis of causes. Measures against evil. Statistical data). Bulletin n. 7, 1936.

37. STROMQUIST, Nelly P. Latin American education in globalized times.

38. SÃO PAULO. Secretary of Education and Public Health. Board of Education.

39. VIRGINIO, Alexandre S. Education and democratic society: sociological interpretations and challenges to the political education of educators. Sociologies, v. 14, no. 29, p. 176-212, 2012.

40. SOARES, José Francisco; XAVIER, Flávia P. IDEB educational and statistical assumptions. Education & Society, vol. 34, no. 135, p. 903-923, 2013.

41. UNESCO. World Declaration on Education for All: Meeting basic learning needs. Jumtien: UNESCO, 1990.

42. ROSE, Nikolas. Powers of freedom: reframing political thought. Cambridge: CambridgeUniversityPress, 1999.

Received: 9 Dec. 2020.

31. PORTER, Theodore M. Trust in numbers: The pursuit of objectivity in science and public life. Princeton: University of Princeton Press, 1995.

32. ROSE, Nikolas. Governing by numbers: figuring out democracy. Accounting Organizations and Society, v. 16, no. 7, p. 673-692, 1991.

33. SOARES, José Francisco. José Francisco Soares: IDEB in the law? Simon's Site, 13 Jul. 2011. Available at https://www.schwartzman.org.br/sitesimon/?p=2352

Sociologies, v. 14, no. 29, p. 72-99, Jan./Apr. 2012.

38. TEIXEIRA, Anísio. Public education: administration and development. Rio de Janeiro: Officina Graphica of the Department of Education, 1935.

Accepted: 15 Mar. 2021.

Machine Translated by Google