Assessment

1 Assessment overview

This course will be assessed on the basis of a Portfolio consisting of a Reflective Journal and a Group Project.¹

The Portfolio is the sum of weekly self-reflections in the Reflective Journal, the feedback you will receive on your reflections, the Group Project and the Final Self-reflection. The course is assessed based on all of these components. There are no weights for the different components, so it is important you engage with all of them.

Learning adjustments

Students with learning adjustments (whether these are in place from the beginning of course or during the course) should get in touch with me (Stefano) as soon as possible to discuss the assessment format and make adjustments if needed.

The following sections give more details on the Reflective Journal and the Group Project.

Important

Note that if you are in touch with your student adviser and/or the Teaching Organisation, we are usually not informed about it, so if you made arrangements with them while the course is in progress, please let us know.

1.1 Formative tests

You will have the chance to test yourself with two formative assessments. These will be online tests with multiple choice and true/false questions. Formative assessments do not contribute to the course mark and they are meant to give you an idea of areas of strength and areas for improvement.

2 Reflective Journal

Overview

• You should post your self-reflection each week.

• You should post your self-reflection by the Wednesday following the week the self-reflection is on. This is not a hard deadline and it is fine if you have to catch up with your self reflection in a later week. If we find that you have not posted in a couple of weeks we will get in touch.

• You should write about 100-200 words for each self-reflection. This is not a hard target or limit. Sometimes you might need to write a bit more or a bit less, as long as you cover the important points. You are also given the space to write more than 200 words, but we might not be able to comment on everything extra to keep our work-load manageable.

Learning is an active process: it happens best when we’re all engaged. Learning is also something that we can learn how to do better. Think of a hobby you might have or a skill you’ve learned: you’ve probably perfected it for a while, paying close attention to what did or didn’t work.

That’s why both students and the lecturer (Stefano) will reflect on how the course is going and what they’ve learned. After each class we’ll all write short self-reflections in the Reflective Journal. The tutors will comment on yours and you’ll be able to comment on mine.

Each of us will have our own Reflective Journal on Learn. The tutors and I will be able to see your Reflective Journal, but you won’t be able to see other students’ Journals. My Journal (the lecturer’s) will be visible to all and you will also be able to comment on it.

In your self-reflection, you want to focus on your learning process: what did you learn, how easy or difficult was it, how did the learning affect your views on something? Here are more specific questions you can follow, but feel free to make adjustments. We recommend to lay out your the self-reflection based on the questions (see examples below) and to add a title to the self-reflection in the format “Week X” where “X” is the week number.

Self-reflection questions

• Participation and Engagement
  How have I contributed to class activities and managed my independent study? Am I satisfied with my approach so far?

• Learning and Challenges
  Which new topics did I encounter this week, how did I approach the weekly Challenge, and what difficulties—if any—did I face? How did I address them?

• Impact on Perspectives
  How have this week’s lessons influenced my views on quantitative methods and related areas? Have my perspectives changed, been reinforced, or stayed the same?

• Intellectual Growth and Achievements
  In what ways have I developed academically this week? What am I particularly proud of?

• Extra thoughts (optional; you can write as much as you like here, beyond the 100-200 words, but we might not be able to comment on everything to keep our work-load manageable).

The following boxes show three strong examples and three weak examples of self-reflections.

Strong example 1

Week 1

This week, I actively participated in group problem-solving and contributed ideas in class discussions. My independent study was focused, though I wish I had dedicated more time to reviewing lecture notes before the Challenge. I encountered regression diagnostics for the first time and approached the Challenge by breaking it into smaller tasks. I struggled initially with interpreting residual plots but resolved this through peer discussion. The week’s content reinforced my appreciation for quantitative methods, especially in identifying model limitations. My main academic growth was in learning to critically evaluate statistical assumptions. I’m particularly proud of explaining a tricky concept to a peer, which clarified it for both of us.

Strong example 2

Week 1

I’ve been consistently present in discussions, asking questions that push me beyond surface understanding. My self-study this week was more structured than before, using a mix of readings and practice exercises. Learning about different summary measures was new to me, and I tackled the Challenge by first reviewing the theory, then applying it to the dataset. Although I initially misunderstood one step, seeking clarification from the tutor helped. This reinforced my belief that statistical tools are most powerful when used critically. I’ve grown in my ability to connect concepts across topics, and I’m proud of having completed the Challenge which was very difficult for me. Next week, I aim to experiment with visualising results more effectively.

Strong example 3

Week 1

In class, I’ve been more reserved this week but contributed meaningfully during smaller group work. My self-directed study was less consistent than I’d like due to other commitments. The new topic was Bernoulli regressions, which I approached with online tutorials and textbook exercises before attempting the Challenge. I struggled with interpreting logit estimates but improved after re-reading lecture slides. This week slightly shifted my view of quantitative methods—I now see how easily results can be distorted if assumptions are ignored. My biggest intellectual gain was in developing a check-list for choosing models. While I’m proud of completing the Challenge despite time pressure, I’ll need to manage my schedule better to stay engaged in discussions.

Weak example 1

I did my work and took part in class. I learned some new things. The Challenge was okay and I finished it. My views on statistics are the same. I have grown a bit and I’m proud I completed my work.

Why it’s bad: No specifics, no examples, no reflection on challenges or growth.

Weak example 2

Participation: I was in class. Independent study: I read. New topic: Regression. Challenge: I did it. Views: No change. Growth: Learned more stats. Pride: I finished on time.

Why it’s bad: Merely answers prompts without linking them or showing thought.

Weak example 3

This week I came to class and listened to the lecture. I joined in when people were talking, although I didn’t say a lot. I did some reading at home but not as much as I probably should have. We learned a new topic about statistics, which was interesting but also quite hard to follow. I did the Challenge activity and submitted it on time. It was a bit tricky but I figured it out eventually. My view on quantitative methods hasn’t really changed; I still think they are useful but complicated. I guess I’ve learned a bit more about how to use them, but it’s hard to say exactly what. I’m proud I got everything done, although I didn’t always understand it fully. I didn’t ask for help because I thought I could manage on my own.

Why it’s bad: Although it meets the word count, it stays vague, avoids specifics, and offers no meaningful insight into participation, learning, or challenges. It lists activities without explaining what was learned, how difficulties were addressed, or how the week’s work connects to personal growth. It also misses opportunities to show critical thinking or outline plans for improvement, making it more of a log than a self-reflection.

2.1 Instructions for Weekly self-reflections submission

After every class you’ll need to write one self-reflection in the Reflective Journal on Learn. Afterwards you need to read our feedback on your entries.

What happens is:

1. We have class.
2. We reflect on our learning process.
3. We write down a self-reflection in one entry in the Reflective Journal on Learn.
4. The tutors comment on your reflections as soon as possible, and you might comment on mine.
5. We might also give general feedback in a collaborative discussion in class or via Learn announcements.

3 Group Project and Final Reflection

Deadlines

• You must submit a Project Proposal to Turnitin by Thursday 6th November at noon on Learn > Assessment.
• You must submit your Final Reflection, including the product of your Group Project to Turnitin by Thursday 11th December at noon on Learn > Assessment.

3.1 Overview

As part of the assessment you will have to work on a Group Project. This project can be anything (some ideas below) so the structure of what you will have to submit differs depending on what you end up doing.

Independent of the type of project, you will have to write a Final Reflection where you propose and justify a mark for yourself based on your experience with the course and with working on you Group Project. Note that your mark suggestion is just a suggestion and the tutors and I will make the final decision. You should follow the marking rubric in Section 4.1 below. This reflection is what you submit to Turnitin on Learn by the deadline of Thursday 12th December. The reflection should contain the “product” of your Group Project, either as text following the reflection or as a link (since the Group Project can be any of different types of products).

The requisites for the Group Project are the following:

• It has to be about linguistics, i.e. research on Language and/or languages.

• It has to be on knowledge-oriented (aka “basic”) research, not on application-oriented (aka “applied”) research. Language/speech technology topics are not allowed.

• It has to be in or on R and or Stan (used by R to run Bayesian regression models). Other programming languages are not allowed.

Ideas for Group Projects

• Pick a linguistic question that interests you and run a mini quantitative study (this could well be a study where you also act as participants). Write a data analysis report.

• Find a linguistic topic and collect published effects from the relevant literature (the first step towards a fully-fledged meta-analysis). Curate the data set and make it available on the Open Science Framework and/or GitHub.

• Conduct a meta-analysis of a topic in linguistics based on existing literature.

• Design exercises and/or tutorials for the course using linguistic data.

• Write a collection of sonnets on the replicability crisis.

• Write an essay on a minoritised statistician.

• Choreograph an interpretative dance that illustrates how coefficients of a model of lexical decision task data are estimated with the Markov Chain Monte Carlo algorithm in a regression model.

• Rap a song about the history of statistics.

Important

Note that the project has to be on a “knowledge-oriented” topic, rather than on a application of statistics to practical problems or language processing/technology.

So for example the following project is fine: “Do participant respond faster when listening to synthetic speech generated by algorithm A vs those generated by algorithm B?” But the following projects will not be appropriate for the Group Project: “We want to write a speech-to-text algorithm in R” or “We will develop a natural language virtual assistant for online shopping” or “I will train a forced-aligner model for a new language”.

As explained above, you will have to submit a Final Reflection including the product or link to the product of the Group Project by by the deadline of Thursday 11th December at noon.

For any question about assessment, post your question on Piazza (unless it’s of a sensitive nature, then get in touch with Stefano).

Warning

Each project will be different, so if you are unsure about how to approach your project idea or if you need help finding an idea in the first place, please get in touch with Stefano!

Stefano does not have telepathic powers (yet) and the only way for him to help you is for you to reach out. :)

3.2 Instructions for Project proposal submission

Deadline

• You must submit a Project Proposal to Turnitin by Thursday 6th November at noon on Learn > Assessment.

• You will have to submit a very short project proposal for approval by Thursday 6th November to Turnitin on Learn, but the earlier you submit the earlier you can start working on it! We will be checking submissions as they arrive and ask to get in touch if there are any issues.
• Only one person per group should submit the proposal (there is no need for everyone in the group to submit the project proposal and you should just add all the names of the group members).
• A few sentences explaining the project will be sufficient. Note that this deadline is an informal deadline and there is no possibility of getting an extension.

Important

If your project requires you to collect data from participants who are not students in this course, you will need to abide to a generic ethics approval sought by Stefano for this course. This means there will be limitations on which participants you can gather data from and which data you can collect from them. More information TBA.

The following boxes give you some examples of possible projects and project proposals, with an explanation on how you can go about completing these projects.

Example project 1

Project proposal

We want to replicate the study by Keogh, Kirby, and Culbertson (2024). The study use an artificial language learning paradigm to test the relationship between working memory and regularisation. We will run the exact same experiment with 5 participants and conduct the same analysis they did.

How to complete it

This is a replication project. You will need to understand the original study design, make sure you have the same materials and protocols and you will have to recruit 5 participants (these can be students from the QML course, including yourselves). After data collection, you should analyse the data as in the original study and write a short report focussing on the analysis approach and results. Specifically, you should discuss how the new results compare with the ones from the original study.

Example project 2

Project proposal

We want to collect recordings of ourselves reading a passage to illustrate different accents. We will include accents from: Scotland, England, India, Tanzania and Hungary. We will curate the data by formatting it so that it’s FAIR (Findable, Accessible, Interoperable, Reusable) and we will make it available on the Open Science Framework with a CC-BY license. The data will be thoroughly documented.

How to complete it

This is a data curation project. You will collect, format, organise and document data which you will make available online. This requires learning about file data management, file formats, Open Research practices and principles.

Example project 3

Project proposal

We want to draw an abstract paining that represents the analysis approach used in Hampton and Coretta (2024). The painting will be accompanied by a written piece with an explanation of the approach, our understanding and how this has shaped the painting.

How to complete it

This is creative project. You will have to carefully read the paper, understand the analysis, which will likely be of a more advanced level compared to what you have learnt so far, and use your creativity to create a painting (no need to be artists to make a painting!). The written piece will have to show that you have understood and internalised the analysis and that this understanding has driven the production of the painting. What you submit is the written piece and high resolution photo of the painting.

Example project 4

Project proposal

We are interested in the relationship between vowel height and vowel duration in Hungarian. We will run a small-scale study with recordings from 2 or 3 speakers of Hungarian. We will analyse the vowel duration using Bayesian regression models.

How to complete it

This is research project. You will have to think of a research question/hypothesis (they do not have to be novel and you can reuse existing questions/hypotheses) and collect and analyse data. Note that we are restricted by a generic ethics approval (see above, more information TBA). The written report will have to focus on the data collection and analysis (you should put less focus on literature review and discussion).

Example project 5

Project proposal

We want to re-analyse the data from Ota (2013). The original study used frequentist regression models, but we will instead apply a Bayesian regression model.

How to complete it

This is robustness assessment project. You will have to get hold of the original data and re-analyse it using a different approach than that in the original study. The report will describe the original approach and your approach, report the results and assess how these are different (if at all) from the original study’s results.

3.3 Instructions for Group Project and Final Reflection submission

Deadline

• You must submit your Final Reflection, including the product of your Group Project, to Turnitin by Thursday 11th December at noon on Learn > Assessment.
• Normal extensions are not possible, so carefully plan your time accordingly. Learning Adjustments extensions and Special Circumstances applications are possible. Note that Learning Adjustments extensions do not extend to group-mates who don’t have Learning Adjustments in place.

• You should submit a PDF file.
• You should use this Quarto document template (right-click and download). For the submission, please render the Quarto file to PDF (I suggest trying this as soon as you download the file, to ensure you have all the necessary software for rendering to PDF).
• Every student should submit. Your submission should include the same Group Project output or link to it (we will discard the similarity ratings on Turnitin).
• In your submission, include the Group Project title and the NAMES of the members of the group (not the student number nor the exam number), including yourself, as per the template linked above.
• If you are unsure about anything, please post a question on Piazza.

4 Feedback and marking

Feedback will be provided to you (1) during class, (2) as comments on your weekly self-reflections and (3) during office hours (it is up to you to book meetings with me; you can do so here: https://bit.ly/33BH84L), (4) on the Final Reflection + Group Project submission.

4.1 Marking rubric

The following marking rubric will be used for the final mark of this course. It is an adaptation of the Psychology/LEL Honours marking rubric. Note that within each mark band, you can only get the 5’s (so 65, 75, 85, etc…), except for 92.

Grade	Understand general principles of data analysis	Develop state-of-the-art Open Scholarship practices	Conduct data analyses with R
A1 92 Excellent	Outstanding in every respect. Shows creative, subtle, and/or original independent thinking. Demonstrates breadth of knowledge and deep understanding of summarising, visualising, and modelling data.	Outstanding in every respect. Demonstrates breadth and depth of understanding of Open Scholarship principles. Shows creative, subtle, and/or original application, with critical and insightful evaluation of transparency, reproducibility, and responsible practices.	Outstanding in every respect. Demonstrates breadth and depth of understanding and use of R. Applies methods creatively and accurately, showing original and sophisticated independent thinking.
A2 85 Excellent	Outstanding in some respects. Shows original, sophisticated independent thinking. Demonstrates a thorough understanding of summarising, visualising, and modelling data.	Outstanding in some respects. Demonstrates a thorough understanding of Open Scholarship principles. Shows original and sophisticated application, with insightful analysis of responsible practices.	Outstanding in some respects. Demonstrates a thorough understanding and use of R. Applies methods correctly and effectively, showing original and sophisticated independent thinking.
A3 75 Excellent	Very good or excellent in most respects. Explores summarising, visualising, and modelling data fully. Shows some complex and/or sensitive independent thinking. Demonstrates a sound understanding.	Very good or excellent in most respects. Explores Open Scholarship principles fully. Demonstrates a sound understanding and shows some complex and/or sensitive independent thinking in application.	Very good or excellent in most respects. Explores use of R fully. Demonstrates a sound understanding and shows some complex and/or sensitive independent thinking in application.
B 65 Very Good	Good or very good in most respects. Demonstrates a good understanding of summarising, visualising, and modelling data. Provides good synthesis, analysis, and evaluation. Concentrates on the main issues.	Good or very good in most respects. Demonstrates a good understanding of Open Scholarship principles. Provides good synthesis, analysis, and evaluation. Concentrates on the main issues.	Good or very good in most respects. Demonstrates a good understanding and use of R. Provides good synthesis, analysis, and evaluation. Concentrates on the main issues.
C 55 Good	Clearly meets requirements. Shows evidence of sufficient knowledge and understanding of summarising, visualising, and modelling data. Demonstrates limited critical analysis and evaluation.	Clearly meets requirements. Shows evidence of sufficient knowledge and understanding of Open Scholarship principles. Demonstrates limited critical analysis and evaluation.	Clearly meets requirements. Shows evidence of sufficient knowledge and understanding and use of R. Demonstrates limited critical analysis and evaluation.
D 45 Pass	Meets minimum requirements. Demonstrates sufficient knowledge and understanding of summarising, visualising, and modelling data at a basic level. Lacks detail, elaboration, or explanation.	Meets minimum requirements. Demonstrates sufficient knowledge and understanding of Open Scholarship principles at a basic level. Lacks detail, elaboration, or explanation.	Meets minimum requirements. Demonstrates sufficient knowledge and understanding and use of R at a basic level. Lacks detail, elaboration, or explanation.
E 35 Marginal Fail	Does not demonstrate sufficient knowledge and understanding. Work is too limited or inaccurate. Provides poorly developed or incoherent account.	Does not demonstrate sufficient knowledge and understanding of Open Scholarship. Work is too limited or inaccurate. Provides poorly developed or incoherent account.	Does not demonstrate sufficient knowledge and understanding or use of R. Work is too limited or inaccurate. Provides poorly developed or incoherent account.
F 25 Clear Fail	Very weak or shows a decided lack of effort. Displays very poor or confused knowledge and understanding. Presents no coherent account.	Very weak or shows a decided lack of effort. Displays very poor or confused knowledge and understanding of Open Scholarship. Presents no coherent account.	Very weak or shows a decided lack of effort. Displays very poor or confused knowledge and understanding or use of R. Presents no coherent account.
G 25 Bad Fail	Extremely weak. Displays no knowledge or understanding. Work is incomplete, muddled, or irrelevant.	Extremely weak. Displays no knowledge or understanding of Open Scholarship. Work is incomplete, muddled, or irrelevant.	Extremely weak. Displays no knowledge or understanding or use of R. Work is incomplete, muddled, or irrelevant.
H 0 Bad Fail	Work is of very little consequence, if any, to the learning outcome. Incomplete in every respect.	Work is of very little consequence, if any, to Open Scholarship. Incomplete in every respect.	Work is of very little consequence, if any, to R. Incomplete in every respect.

References

Hampton, Jessica, and Stefano Coretta. 2024. “Language Practices of Emilian and Esperanto Communities: Spaces of Use, Explicit Language Attitudes and Self-Reported Competence.” Journal of Multilingual and Multicultural Development, October, 1–26. https://doi.org/10.1080/01434632.2024.2413933.

Keogh, Aislinn, Simon Kirby, and Jennifer Culbertson. 2024. “Predictability and Variation in Language Are Differentially Affected by Learning and Production.” Cognitive Science 48 (4): e13435. https://doi.org/10.1111/cogs.13435.

Ota, Mitsuhiko. 2013. “Lexical Frequency Effects on Phonological Development: The Case of Word Production in Japanese.” In, edited by Marilyn May Vihman and Tamar Keren-Portnoy, 415438. Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9780511980503.019.

Footnotes

1. Some of the text on this page is from Itamar Kastner’s Morphology course site.