Project: Audio Visual Language Learning App

Link to Starter Kit (initial code).

In this project, you will be developing an audio-visual Language Learning app for the endangered Canadian Indigenous language, Blackfoot. This language has an oral tradition, and therefore this app will show images to the user while playing audio, to help the user associate word sounds to objects such as apples, child, etc. This project was previously developed in collaboration with Dr. Eldon Yellowhorn in the Indigenous Studies Department, and the Peigan Board of Education in Lethbridge, Alberta.

This project is broken down into 2 parts. See assignments page for dates.

  • Part 1: develop the module draw.py with image processing functions.
    • A sample solution for draw.py will be posted on the day after the last day to submit part 1.You can use this sample draw.py for your Part 2 if you like.
  • Part 2 is the complete application.

You may do either or both of the parts solo or in a pair.


Part 1: Image Processing Module

First download the Starter Kit. It contains images, sounds, and testing functions. Then implement the following 5 functions in a custom module called draw.py.  The starter kit includes a skeleton implementation of the necessary functions to get you started.

  1. recolor_image(img, color)
    Changes all non-white pixels of the image referred to by the parameter img to the specified color, and returns a new image with the changes.
    • A pixel is non-white if any of its RGB values are less than 240.
    • img is a  2D array of RGB pixels as seen in class to represent images.
    • color is a list containing 3 integers, each from 0 to 255, representing RGB values.
    • This function must return a new image; it must not change img. Use cmpt120image.get_black_image() to create a new canvas (image) that your function can return.

  2. minify(img)
    Returns a new image which is half the size (half the height, half the width) of the image img.
    • For this function, you can assume the width and height of images are even.
    • Don't change img, you must create a new image to return.
    • Hint: The average of each 2x2 block of pixels from the original image becomes one pixel in the result image.
      • As an example, the R/G/B values of the pixel at result[0][0] with have the average-R, average-G, and average-B values of the pixels at original[0][0], original[0][1], original[1][0], and original[1][1].
      • Start by thinking about each pixel in the smaller image, and then think about what pixels from the larger image to use to compute that pixel. If stuck, draw it out on paper and write down the first 3 or 4 pixels to find a pattern.

  3. mirror(img)
    Returns a new image which is a left-to-right flipped copy of the image img.
    • Don't change img, you must create a new image to return.

  4. draw_item(canvas, img, row, col)
    Modify canvas by drawing the non-white pixels (R, G, or B value < 240) of the img onto canvas.
    • canvas and img are 2D images of RGB pixels. This function should assume that the canvas image is large enough to contain img at the row col indicated; row and col do NOT need to be validated.
    • row and col represent the top left pixel of canvas where the top left corner of the img should be line up.

  5. distribute_items(canvas, img, n)
    Draws img onto the canvas n times in random locations.
    • Overlapping item images is OK.
    • Hint: Consider calling one of the functions above.
    • Ensure that each copy of img is fully visible inside canvas (i.e., not off the edge).
    • This function will not change the colour, size, or content of the img: it just places n copies of img into the canvas at random locations.

To test functions 4 and 5, you'll need to create a canvas corresponding to a white background of a reasonable size (such as 300 (h) x 400 (w) pixels), using the provided function in cmpt120image.py.

Functions and their images

Testing Part 1

See here for directions on testing part 1.

Submission for Part 1

  • Submit to CourSys
  • You will need to create a group in CourSys to submit. If you worked solo, create a group for just you. If you worked with a partner, create a group in CourSys and invite your partner. Only one person per group needs to submit.
  • Ensure you have done the following:
    • Added the names of all group members to the top of draw.py.
    • Removed any debugging comments or extra code in draw.py.
  • Submit a zip file of your full project folder, including draw.py and any test code and images.
  • We will likely mark using not only the provided test code, but also additional testing routines, so ensure that you have not changed the names or parameters of the required functions in draw.py.


Part 2: Language Learning App

Part 2's application has 3 main parts:

  • Learn: Images are shown to the user while listening to the audio.
  • Play: The application will play the audio for a word, then show the user images of several items. The user counts how many of the images corresponding to the spoken word they see and enters the number. They get feedback on if they were correct.
  • Settings: By default, the app will teach 3 words at a time, but the user can change the number of words being taught.

Here is a video depicting an example of the flow of this program: Blackfoot Project demo video (YouTube)

Part 2 makes extensive use of the draw.py module from part 1. You may use your own draw.py, or the provide sample solution (posted after part 1 is due).

App Features

The items (words) that the app will teach the user in this game are listed in a provided blackfoot.csv file in the Starter Kit. Some examples are apples, bread, burger, child, etc.

  • Your application must first read this .csv file containing all possible items/words.
  • For each word "xyz" in the .csv file, there is an image images/xyz.png and a sound sounds/xyz.wav.
  • When marking, we may add/remove words (and provide the correct images and sounds). You cannot hardcode any of the words nor file names. Everything has to depend on the words in the .csv file.

Menu

The app will first show a menu to the user to choose if they want to Learn, Play, or change Settings. 

MAIN MENU
1. Learn    - Word flashcards
2. Play     - Seek and Find Game
3. Settings - Change Difficulty
4. Exit
<Text> Choose an option:

Note on input()

To make the application work with or without the image window being shown, you should use the provided read_input_with_screen(prompt) function instead of the usual input(prompt) function. This new function will behave similar to the usual input() function except that it allows our image window to still be moved around the screen.

If there is no image window being shown, it will print out the prompt with <Text> at the front, indicating the user should type their response into the text console and press enter just like normal.

If there is an image window being shown, it will print out the prompt with <Screen> at the front, indicating the user should click on the image window and then type their response and press enter. As you type nothing will be shown in either the image window or the console.

1. Learn

  • The application will cycle through all the items to learn once.
  • For each item to be learned, it will randomly place one image of the item on the screen and play the sound of its name.
  • The user presses enter to continue to the next item to be learned.
  • When the program first starts, only the first 3 items in the .csv file are taught to the user.
  • The user can change this number in the Settings menu, up to the number of items in the .csv.
  • Hint: Which of your draw.py functions can help place one copy of an image somewhere on a screen, randomly placed?
  • Hint: Your program will have to create a blank image (white canvas) and draw the image into it and show it. The cmpt120image.py module can create a black image; you'll need to write code to make a white image.

2. Play

This is a seek-and-count game, where the user has to listen to a word, and count how many of that item are in the image.

First, the user can specify how many rounds they’d like to play. Then, for each round, the program must:

  • Create a challenge list of 3 words from the list of items the user is learning (defaults to 3).
    • You must ensure that the same word is not repeated in this list of 3 randomly chosen words.
    • Hint: You may want to explore the random.shuffle() function.
  • For each item in the challenge list:
    • Recolour the image to a random colour.
    • Randomly decide to minify (or not) the image.
    • Randomly decide to mirror (or not) the image.
    • Choose a random number n between 1 and 4, and display n of that item in the canvas. 
  • All images for a single item will have the same properties.
    • For example, if a randomly chosen item is dog, the random colour is blue, the number randomly chosen is 4, and it is randomly decided to minify, and randomly chooses to mirror, then all 4 of the dog images should be blue, half sized, and mirrored on the canvas.
    • One specific item (e.g. dogs) may have more than one image processing applied before placing all copies in the canvas; i.e. it's possible for the dog image to be both minimized and mirrored.
    • The random colour can be from a list of colours you have defined, or randomly come up with (though, ensure the colour is not too close to white or we won't be able to see it).
    • For randomly minifying or mirroring, do so with a 50% probability for each alteration.
  • Ask the user how many of the items they found. If they enter the number correctly, then a message confirming that it is correct should be printed to the user in th console, otherwise inform them that it is incorrect. 
  • To play audio, call the function provided playSound(soundfile).

3. Settings

By default, the player learns 3 words. They can change the number of words to learn and or to play using the Settings option in this menu. The application should NOT allow the user to change the number of words to less than 3 or more than the number of words in the blackfoot.csv file.

You must not have any global variables. You cannot have the list of words form the CSV file be global. You cannot make the number of words to teach global.

Suggestions:

  • Create a function for each of the three large sections of the app.
  • For the settings, pass in any information you need for it to work.
  • Return the selected number of words to learn from the settings function.

Submission for Part 2

  • You will need to create a new group to submit because groups from part 1 will not automatically carry forward to part 2. If you worked alone, it will be just you in the group. If you worked together, invite your partner. Only one person per group needs to submit.
  • Ensure you have done the following:
    • Added the names of all group members to the top of your code.
    • Removed any debugging comments or extra code.
  • Submit a zip file of your full project folder, including draw.py, your main application and images/sounds.


Getting Started

Download the Project Starter Kit from link at top of page.

General suggestions

  1. Start Early: It usually takes longer than you expect to design, implement, and test your code.
    • Plan to work on it over multiple days.
    • When done for the day, add a TODO list in your comments about the next step.
  2. Arrange Group: If working with someone else, arrange with your teammate when you will meet, how you will communicate, what is important for you in teamwork.
  3. Read Description: Read the full description!
    • Review the sample images and understand the problem.
  4. Plan Solution: Plan out how you will do what is required. Don't code yet!
  5. Gradually Implement: Write a little code, test a little code. Work on one function at a time and get it working. Use the provided testing file to help. Step through the code with a debugger; add print statements if needed; carefully check your loops!
    • Start with the draw_item(...) function because it's what the test code needs first!
  6. Submit: Submit your work to each part on time.
  7. Discuss: Discuss the problem with your team member/s, with the TAs, the instructor, peer tutor sessions. If you have questions join office hours. OK to use the discord channels for questions of interest to the whole class (don't post your code).
  8. Start Early: No really. It will take a lot of time! Get going now!

Placeholder functions

Both the Learn and Play parts include image processing, sound processing, and other various tasks that may be implemented gradually, and/or possibly independently. In general, for coding a program of considerable size, while you have not coded all the needed components and functions you may temporarily define “placeholder functions”. 

A placeholder function would be defined with the appropriate name (e.g. drawing_something(...)) and be called from where it corresponds, but temporarily the only content of the function would be, for example, print("inside drawing_something(...)"). If it is a productive of fruitful function, you would temporarily return a fixed possible value (e.g. return 1, if the function is meant to return numbers for example). At a later stage, or independently, you can develop such functions, and once they work you can incorporate them into your whole code.