Institutional ResearchWhat problems are wetrying to solve?Mantz Yorke & Bernard Longden
Sorting out types of problemsReports that say that something hasn't happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know.
But there are also unknown unknowns the ones we don't know we don't know.
Donald H. Rumsfeld, Department of Defence - news briefing, February 12, 2002
The challenge facing any University is to Be smarter and better manage the known knowns; Inquire into university life to seek out the known unknowns and Reduce the risk of unknown unknowns as they relate to:Demographic changesNational policy driversInternational opportunities
Types of problem :Types of institutional researchWhats going on type of researchWhy is it going on type of research ExplanatoryDescriptiveWithout theory, research is impossibly narrow. Without research, theory is mere armchair contemplation. (Silverman, 2001: 294) Two broad types of research
Institutional research - four facesVolkwein (1999) has provided a useful analysis of the aims of IR The four Faces of IR.Research for improvementInformationEducational researchAudienceRoleAdministrative,InstitutionalFormative & internal for improvementSummative & externalAccountabilityAcademic,ProfessionalPresentation
Terenzinis organisational intelligenceHierarchy of complexityFactual knowledge analytical skillsMajor issues or decision areas facing the universityCulture knowledge of what a college is and where it has come from
What problems are we trying to solve?The view from inside - BLVolkwein (1999) has provided a useful analysis of the aims of IR The four Faces of IR.InformationAudienceRoleAdministrative,InstitutionalFormative & internal for improvementSummative & externalAccountabilityPresentation
Administrative/Formative-known knownsIR as Information Authority Administrative formative and internal Emerging within institutions as:
Information officeManagement informationPlanning and developmentInstitutional ResearchPrimary functions:
Clarifying the known knownsDescribing the university in data termsEducating the university community about itself
What problems are we trying to solve?Information authorityBase data about the institution profile and fact bookStatutory returnsValidating data is this really us?Size and shape of the institution to assist in setting targets and plansTerenzinis organisational intelligence level Technical/analytical intelligenceIn the UK these are well developed possibly over load the university with dataThe challenge is to reduce the plethora of tables and cross tabs to identify Key Questions that are crucial to the organisation for success
What problems are we trying to solve?Presentation facilityMarketing the best aspects advocate roleAudit and external accountability Performance measureLeague tablesTerenzinis organisational intelligence level Technical analytical intelligenceSpin and managing data Political pressure to maximise impact of dataDenial about the message data givesRisks and concernsPrimary functions:Conform data to agreed descriptors Performance IndicatorsSatisfy public funding legislative requirements
Student Data flow Student data Students Aggregate early return to HEFCEData informs funding - audit Individual Summative activity data return to HESAData passed to HEFCE who recreate December aggregate returnBut same data also used to inform public through: Publications, commodification of data Rankings
Data sources to assist with the questionsUKHigher Education Statistics AgencyPublished data sets HEDIUCASUnistatHEFCE (?)National Student SurveyUSNational Centre for Education StatisticsIPEDS universe institutional census NPSAS National Post Secondary Student aid StudyBPS Beginning Post Secondary - Longitudinal StudyB&B Baccalaureate & Beyond - Longitudinal StudyNational Science Foundation
Some propositions that we might need to addressShould the IR questions be articulated more clearly and data collected in response, or should we continue to collect data and then use what we have for policy decision?Do universities make good use of the data within their own university?Does IR constitute value for money?Is data critical to teachers, administrators, decision makers?Is the emphasis between data collection and analysis appropriate? What lessons can be learned from the research communitys use of data sets in the US?
Comparison sometimes interestingUK HESA and US IPEDS similar objectives and operationUS interest in scientific advancement not reflected in the UK Possibly RAE provides nearest comparisonUS positively encourages Institutional ResearchAssociation for Institutional Research (US) in conjunction with National Centre for Education Statistics provides funded summer school to encourage analyses of public dataUK implicitly discourages Institutional Research HESA charge for the use of our data!
What problems are we trying to solve?The view from outside - MYResearch for improvementEducational researchAudienceRoleFormative for improvementSummative & externalAccountabilityAcademic,Professional
Academic/Formative: known knowns and unknownsIR as Research for Improvement
Academic formative and internalAnalysing (afresh?) existing data, or investigating institutionally-focused research questions, with a view to influencing practice.
Problem: such activity is often scattered within an institution, and unco-ordinated.
Research for improvementExamples:
Student feedback, including NSSOutcomes of quality assuranceAssessment results, cross-tabulated with demographics, subject disciplines, etc.Incidence of plagiarismResearch assessment exerciseWhat problems are we trying to solve?
Academic/Summative: known unknownsIR as Educational Research Academic summative and external, though with some formative and internal relevanceInvestigation of issues which are of wider relevance than to a single institution, but which may be triggered by an internal institutional need.Evidence needs to be particularly robust.Potential for dissemination of findings beyond the individual institution, in which case criteria for peer review may well apply.
Pedagogical innovations of various kindsAssessment of achievement in workplace settingsVariation in honours degree classifications across subject disciplines and institutional typesRelation of PT employment to achievement on FT programmesStudents experiences of HE (FYE; PT; HE in FE, etc)Non-completionWhat problems are we trying to solve?
In the end what answer can we give to Gnothi Sauton Know thyself?
*In 2003 I was a regular listener to Radio 4 Broadcasting House with Eddie Mair.He had a regular spot where he showcased the saying of Rumsfelt.At first this amused but then I realised how profound the statement was.
*Thinking about an agenda For Institutional Research in the UK and using the Rumfeld categories then our task must surely be to Be smarter and better manage the known knowns
Inquire into university life to seek out the known unknowns and reduce the risk of unknown unknowns*Descriptive research what going on Explanatory why is it going on.
Institutional research has both elements Volkwein has a helpful way of stratifying the nature of inquiry for institutional research*The matrix provided by Volkwein(1999) represents the two dimension of AudienceandRole We plan to make use of this categorisation of IR.I will be looking at the from the internal aspects and Mantz will focus on the external cells*We have layered the Volkwein model with that of Terenzi who has developed a hierarchy or tiered model for the level of knowledge and skills required to manage the various operations within IR.The basic level required is that of an ability to provide technical and analytical skills to bear on a research topic. There is a essential requirement to have a sound understanding of the factual data elements floating around the university. Some universities are hampered by the idea that Knowledge ensures control and therefore the university may be unwilling to encourage too many people to have this level of intelligence.Building on the base layer is the issues intelligence layer where it is essential to have an understanding of the dominant prevailing issues that require a reaction by the university. There may be a disparate body of people within the university capable of operating at this level but without access to data and a ability to overcome the various barriers erected then an opportunity to maximise the known knows is lost.The highest level of contextual intelligence occurs when data across the sector is brought to bear on sector wide problems.*Mantz and I have agreed to use this matrix as a useful means of providing a focus on our different interests in institutional research.In broad terms a division in terms of internal, institutionally specific focus issues and those that are more generic and external with a sector interest and focus.First then what are the problems that focus those of us with an internal perspective?Over forty years it is possible to discern an evolution of IR to its present day function within the higher education landscape.In comparison with the US here in the UK we still have a long way to go.I plan to make direct comparisons with the US in my part of the presentation.First a review of internally focused institutional research from the Volkwein perspective*The development of IR as an Information Authority is an ongoing project. When comparing to the US it is clear that here in the UK the approach has been different. Iin the US there are clearly defined operational units termed IR Offices in most if not all universities.In the UK the functions might be covered but in disparate offices such as finance office or the estates office or the registry. Pulling all these different data sets together presents an enormous technical challenge. The primary function is to make known the known knowns to the university community.So what is are these known knowns that the community ought to know about what are the problems we are trying to resolve?*I have introduced the useful hierarchy developed by Terenzi to order organisational knowledge.In his analysis he identifies three tiers of organisational intelligence.Technical and analytical intelligence Issues intelligenceContextual intelligence
At the base of the pyramid and requiring the lest width of knowledge is the Technical The challenge is to reduce the plethora of tables and cross tabs to identify Key Questions that are crucial to the organisation for success*The main point of this slide to to reinforce the concern that data released by institutions for one purpose funding audit is then used managed by commercial interests to support ranking and league tables data *Differences are interesting in that there are only recently longitudinal studies eg NSS National Centre for Education StatisticsIPEDS universe institutional censussanction relates to federal aid data collected from over 6,6000 universities and collegesmandatorydata about students - demographics, work load, progression data, completion data, financial aid resources- institutional resources, finance, human resources ( salaries, employment status)
Longitudinal studies and sample surveysNPSAS National Postsecondary Student aid StudyNSOPF National Study Postsecondary FacultyBPS beginning Postsecondary Longitudinal StudyB&B Baccalaureate & Beyond Longitudinal Study what happens to the graduates
National Science FoundationInterest in the status of science education in the US*
**The Ancient Greek aphorism "Know yourself" (Greek: or gnothi seauton) was inscribed in the pronaos (forecourt) of the Temple of Apollo at Delphi - according to the Greek periegetic (travelogue) writer Pausanias (10.24.1).The aphorism has been attributed to at least six ancient Greek sages: