Week 1 Assignments

Chapter 1 - What is Evaluation

1. I am going to evaluate a program we host at the public library. Currently, I am planning on evaluating the library's Summer Reading program. The goal of summer reading is to encourage kids to keep reading over the summer break to help avoid losses in reading comprehension. During Summer Reading, we have kids account for their reading and activities by marking boxes off of a check sheet. They turn in the check sheet and get entries into the weekly and final prize drawings. I expect more kids to participate than did last year. I also expect them to be excited and reading many books.

2. I will evaluate this program because in years past, we had been doing the Summer Reading program Online. People were supposed to basically do the same thing as this summer except on the website. This was a pain for the families with multiple kids. It was tedious and discouraged participation. My direct supervisor convinced (with great effort) her boss to allow us to go back to the "old school" way. I would like to evaluate the success of the program to show her how much her hard work paid off!

3. Regarding the two definitions of evaluation, I am hard-pressed to pick just one! I think both go hand in hand most often but are necessary at different times. In evaluating a program in the library, we first must make sure that we are meeting our objectives. If we aren't making our goals, we then must decide if we need to change or drop the program. I suppose, it isn't that we are evaluating to make the decision, though. It seems it would be more accurate to say that we want to know if the program is working like it is supposed to. If we find that it is not and make a decision based on the data, we are making the decision after evaluating. So, I will settle on being in the camp of "Evaluation is the systematic process of collecting and analyzing data in order to determine whether and to what degree objectives have been or are being achieved."

Chapter 2 - Why Evaluate?

I've chosen to evaluate the "Rich Kids, Poor Kids" scenario.

1. Cashflow Technologies will benefit from participating in the evaluation project because they will be able to add having been professionally evaluated to their sales pitch. Also, they will probably get feedback on their product. MCCD will get to find out if the program addresses their students' financial literacy needs. A limitation of this evaluation is that Cashflow Technologies may choose not to address the feedback they receive. Cashflow Technologies might also not like the evaluation team having access to their sales data.

2. Because Cashflow Technologies and MCCD both benefit from this evaluation they will have buy-in that will help the evaluation be successful. Cashflow Technologies has been making these materials for a while now, so MCCD has a good reason to believe the program will work. The course was already offered and if it was dreadful, MCCD would definitely have heard about it! The evaluators have access to all the information they could need and have a big budget, so they will be able to complete the evaluation successfully.

3. Cashflow Technologies can use the results of the evaluation to modify and expand on the course as well as all of their other products. Having access to the evaluation results could make all the difference in them being successful when they market the program to other schools. MCCD will be able to use the results to inform their stakeholders that the previous course was a success and that they will continue to have good experience by continuing with the larger program. The students in the EdTech program will get to use the results as proof that they are learning the concepts and techniques of evaluating programs.

Week 2 Assignments

Chapter 3 - Decision Making: Whom to Involve, How, and Why?

A. Apply Gap Analysis to my program. Discuss the program being evaluated with the three "legs".

Leg A = We currently have a space designated for "Maker activities". It is named the Camas Create Space. There are 2 3D printers and a virtual reality machine already inside. Typically, one of the 3D printers is used daily by patrons. The virtual reality machine is only used once most weeks. The printers are bookable by appointment on our website. We have drop-in VR noted on our website. One robot, Cozmo, has been sitting on the counter for a few weeks. It has been used about 5 times, but only because staff suggested that a specific patron use it. This week, I created a presentation for the television to show items available and the goal of the space. I've noticed a couple people noticing it.

Leg B = To close this gap, I will need to plan! I need to design programming for each of the types of equipment we will offer. I need to identify staff members who are willing to participate. I will need to decide how to spend the budget money and then put in the purchase requests to do so. I will need to change the times in the Outlook calendar to reflect the different activities available. I will need to submit programs to someone to put onto the web calendar and to share on our social media. For outreach, I will need to scope out local businesses and connect with staff to find times to offer programming that would benefit their users.

Leg C = I would like to see the Camas Create Space being used for at least two different creative endeavors daily. I want the 3D printers being used by at least 4 different people weekly, the VR equipment being used at least once every week, and the robots being used at least 5 times a week. I want to offer times that staff make themselves available to patrons. I also want to continue offering booked sessions on our website. I want to expand the types of sessions to include our other equipment. I also want to start STEAM outreach to the surrounding locations such as retirement centers, community centers, and pubs.

Chapter 4- Starting Point: The Evaluator's Program Description

This chapter is all about the Evaluator's Program Description (EPD). Creating an EPD will help us track whether a program is meeting the objectives. We create the EPD once we have identified the needs of the program, have a plan for addressing the needs, and have clearly stated the standards of the program. The EPD will help us clearly define the goals and objectives of the program. We will also use it to see what exactly is happening in the program, what and how. The EPD will highlight the measurement tools, too.

Every EPD doesn't look the same. Each of the evaluation recipients will have different objectives of the program. Because what they want out of the program is different, they will have different ways of describing what success looks like as well as the program itself. Creating a different EPD for each of the recipients will help when it comes to evaluating the program; we will know what to measure for each of the users of the evaluation.

The evaluator will need to meet with each of the users of the evaluation. The evaluator uses this time to make sure they have all of the information. They will find out what goals the users have for the program, what kinds of activities they are thinking about having in the program, and any specific standards of measurement must be met.

This chapter also points out the benefits of interacting with the stakeholders at this point. They need to trust you if you expect to get good information from them! The book gives a fun quote from Sir Josiah Stamp of the Inland Revenue Department 1896-1919: "The government are very keen on amassing statistics. They collect them, add them, raise them to the nth power, take the cube root and prepare wonderful diagrams. But you must never forget that every one of these figures comes in the first instance from the village watchman, who just puts down what he damn pleases." I love this because no matter how good my programming plans are, if my coworkers do not "buy in" I will not succeed. We have "sticks in the mud" at the library who just do not want to change things. But, we must. To stay relevant with changing times, libraries have always had to adapt. If, as some people believe, our main goal is keeping books, then we could deny all technology. It wouldn't really serve us well, though. As libraries true desire is to provide all kinds of literacy for every person, we must stay abreast developments.

Chapter 9 - Writing the Evaluation Report

Appendix B - Sample Evaluation Report

As an employee of a city department myself, I identified strongly with the sample program. This sounds a lot like programs we create in the library. The objectives were awesome. When we create programming for teens, especially at-risk teens, we need to make sure to meet more of their needs than just the basic objectives. With this program, the zoo could have only aimed to teach teens to lead the younger kids on a tour. Instead they realize that they need to teach the teens how to properly interact with little kids, how they directly impact the animal world, and that they have worth. The teens reported feeling like they weren't valued by the regular zoo staff. This points out that they were really buying-in to the responsibilities of being a staff member themselves!

I personally like having examples such as this because I haven't run into enough formal evaluation type projects in my working career. I intend to start using evaluations like this for every new project I start.

I was surprised by how many objectives the project had for being such a small scope. It seems that, with great planning, one can really accomplish a lot with little. I bet this project didn't cost much beyond staff time, too! It has such potential for growth! Lots can be done with a "can do" attitude like the director of the zoo has. This program would make a huge impact on the teens.

I had no idea what the data tables were trying to represent, though. I would like to think I am pretty good with statistics and presenting such data, but man, I have no clue how to evaluate the table. I spent a good 15 minutes just trying to decipher it!

Week 3 Assignments

1a

I feel like chapter 2 of the web article was very similar to the book information. The example is quite helpful in further demonstrating a hypothetical situation to compare our projects against. It doesn't seem to provide any new information though. I immensely liked the wording in the goals and the proposed evaluation questions. I've copied them down to reference when I get to that part of my evaluation. As an aside, if they really do budget $75,000 for evaluation no wonder most places don't truly evaluate things!!!

Chapter 6 gives us some great examples of how to acquire qualitative data. I took note of these examples, too. I'm excited to make sure my questions are as thoroughly informative as this source. The data collection plans are fantastically detailed! I suppose with a $500,000 budget, one would need to have a rock-solid plan. I don't feel I've even been on the periphery of an evaluation this rigorous! I'm glad to have read both sources. I feel like the internet source has been helpful with my fleshing-out my project. I don't feel that the internet source can stand alone, though. It doesn't touch on the different types of evaluations one bit.

1b

I'm currently curious about using the Decision-Making Model for my evaluation. I wish the book provided greater explanation of this kind of evaluation. Simply from Table 5.1, I know that many of the words in Intended Outcome are what I am hoping to evaluate. The Sample Evaluation Questions column sound like the ones I was throwing about in my head, too. The Goal-Based Method also sounds promising. Because the book doesn't give enough information about the different evaluation styles, I am going to read further into it to make sure that the Decision-Making Model is the right choice. I will report back any external sources I find helpful.

2a

1. The differences between qualitative and quantitative data are that quantitative data are those that can be expressed in finite numbers while qualitative data are information that must be presented in whole quotes. To gather qualitative data one must ask open-ended questions and take the entire information presented in the answer. Those data are hard to condense into a chart, but are richer with information.

2. The level of data I will collect in my evaluation will mainly be Ration Data. I'm evaluating whether or not the Camas Create Space is used as well as which items are used. I hope to ask some questions of the users such as "which 3D printer do you like better" and "when is the best time for you to use virtual reality". These both fall in the Ordinal Data level of data.

3. Currently the library collects usage data in a SharePoint document. I will develop new tags for the different types of programs we offer. Also, I do not believe there is a way to tag the Camas Create Space as it is a new space. We tag our other rooms, so I will need to add this room as an option. I will also need to develop staff understanding of the importance of logging this room's usage. Right now, although there is a system in place, many of the branch staff are not using it. I will also need to find time to interview patrons using the space. As I am frequently leading programs for new users, I can create a few questions that I can ask during the instruction phase. This will aid in the information gathering being less obtrusive. I won't need to get staff buy-in if I do this by myself, too. It is frequent that staff reports being too busy to do the different statistic gathering tasks assigned to them.

Website review

Site G

I love Khan Academy lessons. They are extremely helpful and cover a plethora of topics. That they are free is truly the greatest part, though. There have been many times throughout college that I've relied on Khan Academy to pass classes that were being poorly taught. These lessons specifically are clearly presented and effectively teach the information. I'd say that these lessons are even good enough to teach statistics to those with no prior teaching on the topic.

Site H

With all the previous site in mind, I'd have to pick Site H as the best. It offers many of the Khan Academy videos in one place alongside other sources. One of the videos I loved the most was What Is Statistics: Crash Course Statistics #1. The examples of why we use statistics bolster desires to learn. I feel like after this specific video, I really feel like I've got a "hold" of statistics as a science and not just "Mean, Medium, and Mode". I love the host of the video, too. This is a great channel! Of course, watching the videos on YouTube forgoes the quizzing option that Khan Academy offers.

Site B

Lastly, I picked Site B as a favorite because it is a cute website for practicing the operations yourself. The font could definitely be larger. The explanations of the different operations are clear but I would place them higher on the page. Also, the way the entire page presents is a little overwhelming at first glance. I enjoyed the chance to try the problems and have them evaluated quickly. The "Help" button doesn't really help but rather answer the question, though.

Evaluation of all sites

• Site A is funny because, although explaining what a statistician does for a living is hard, it is definitely compounded by how ignorant Kelso is! I've never seen his dad before, either, so I personally liked this one.
• Site C was a very simple overview, good for those who just want a quick refresher on the operations.
• Site D offers many neat calculators but the website itself is an eyesore.
• Site E is clean and provides clear definitions of the terms used in statistics. It isn't extremely helpful otherwise.
• Site F was another Khan Academy video shared on YouTube. It's not as good as other lessons they have because the presenter kept saying less accurate words and having to correct himself. This made it harder to follow.
• Site I is cute. I liked the design of the page and the quiz at the end. Other than that, though, it is nothing to write home about.
• Site J was kind of "meh". It was done in a game show style but it isn't better than the other options that provided quizzes such as Site B. Also, I think one of the questions on my run through it was incorrect.

4a

1. My project represents evaluation. I am not looking to answer "why" patrons are using our equipment. I am only looking to see "if" and "when" patrons use the Camas Create Space. I am not attempting to control any variables or to change them. I may try to find out which equipment they prefer to use but I will not seek to find if I can affect their answers by changing variables such as which filaments we offer for the two different 3D printers. I also will not try to establish a cause-and-effect relationship between patrons use of the equipment and what they learn. I am not concerned with whether they "succeed" in using the equipment. I just want to know that we are successfully making the equipment available to users.

2. My project will make use of a judgment sample. I will pick the people that I am teaching how to use the equipment as well as anyone I see using it during my shifts. I expect that this will make the data biased a bit as only inexperienced users will be the one booking my time. I'll definitely look at the numbers of users collected in the SharePoint for the "experienced" users who do not have a staff member assigned to their session, but I will not likely ask them many of the evaluation questions unless I happen to see them working on their project and I have time to interact with them. Our library does a yearly survey of all users and it would be neat to get some info back about our tech offerings through that survey. It seems we ask the same questions every time and I doubt we get uniquely useful information from it.

Week 4 Assignments

1. Chapter 9 - B

Summary

This will be about 5 paragraphs written like the beginning of any paper. I need to give the basics of the project and cover the following parts. I expect to give the explanation of STEAM, Boise Public Library, my branch, what library assistants do (most of this is for the city council, not the library director or my boss). I'll need to mention why we are designating the Camas room as the Camas Create Space and talk about why STEAM programming is good for the community. The end of this section contains an overview of the results.

Description of the Program Evaluated

Program Components

I will cover all of the tech that I buy and what we have currently on-hand. I will talk about patrons use the equipment including booking appointments and staff help. I hope to have other room usage that patrons come up with themselves. Previously we have mentioned that our quilting group should meet in there but of course they have quit coming. There was talk about buying ukuleles, so that is very much on my radar!

Program Objective

The objective of this program is to increase access to STEAM projects for patrons of the Library! at Hillcrest and surrounding area.

• Increase patron utilization of 3D printers.
• Create opportunities for users to reserve virtual reality equipment.
• Start outreach events at nearby retirement home to increase access to STEAM projects.
• Purchase more STEAM equipment and design programming to incorporate its use.

Evaluation Method

Because the goals are mostly to provide more access to STEAM related projects, the methods for evaluating the program will be mostly statistics. I will track usage of the Camas Create Space. I will note when, how often, how long, and for what projects the room is used.

Participants

The participants are library users and people at the retirement home near the library. They will be people of all demographics regarding age, race, gender, sex, and economic status.

There will be staff other than myself who conduct STEAM programming. One is our branch librarian. He is male and has a Master's of Library and Information Science (MLIS). Another is the Lead Library Assistant. She has worked for the Boise Public Library (BPL) for over 20 years. If other staff become interested in the project, I will make sure to include them in the final report. The evaluation will be written for my boss, the branch manager. She has over 20 years of experience in libraries as well as the MLIS. I would like to believe the library director and city council would like to read it, but I doubt they would. I will still write it up for them as audience members. Our director has worked for BPL for very many years. He started in the very bottom position, paging. They are the people who put all the books back on the shelves. He worked up from there. The city council is made up of 4 women and 2 men. Many of them strongly support the library, including the upcoming project to rebuild the main library branch (which has faced open opposition by many people).

Procedures

I will put a sheet for patrons and staff to mark that they used the robots. Staff keeps statistics on the 3D printing appointments currently. I will inquire weekly about whether or not the stats were recorded accurately. When the virtual reality appointments go Online, I will instruct staff to also record those stats the same as the 3D appointments.

Data Sources

At BPL we collect statistics for all of our programs in a Microsoft SharePoint spreadsheet. This makes finding the data extremely quick. As of now, I do not have plans to survey the participants directly. I am mostly only concerned with usage numbers. In the future I will worry about the impact of our programs. If I am able to get outreach at the local retirement home before the project is due, I will probably survey those participants. I will ask questions like "what have you learned from this project", "will you seek out more programs hosted by the library", "what other kinds of STEAM programming are you interested it" and the like.

Results

I see that the final assignment calls for tables! I'm glad: I really love making them and presenting data in them and I just learned how to do the coding in HTML/CSS to make them spiffy. It looks like the tables will mainly make up the appendices.

Discussion

Here we discuss each of the objectives and the results. This gives a chance to talk about the results especially when they point out the bad stuff. I think that there will be a good number of times that our tech won't work like it should, that people will not arrive for their appointments, and sadly the one I worry about most, that staff will not help. We have some employees who just do not want to learn technology. Also, there are one or two staff members who believe they are "just too busy" to help the patrons. This is crazy because that is literally their job. I'm also a bit worried that I won't have the data necessary to discuss! What if no one uses the program materials? I know this won't happen but "what if no one comes" is the typical library programming worry.

Appendix

This will be where my graphs (if they are too big for their page) will go along with any surveys I end up conducting. I believe I should include a screen shot of the SharePoint page to show what kinds of data we collect in our statistics. I'm sure I'll find more things to include as the project progresses.

2. Appendices- C

I loved the report in appendix B. I felt like it was extremely clear. Like I commented today on a classmate's posting, I felt like at first I didn't like the parenthesis comments explaining that the information was skimmed over because the director already knew it. I was worried that the authors should have shown an example that was written for someone else, someone with less internal information. But then, I realized that my own project will be more like this one. It was nice to see that of course I will not need to include the information that my stakeholders already posses. It nicely highlights the point that the different evaluation reports are for different audiences. If my library director and the city council cared, I believe my evaluation report will be very helpful. Also, today my boss's boss asked her for a paragraph or so about what we are doing with the Camas Create Space and STEAM programming in general for her to present to the board of directors! I wonder if she would be interested in reading the evaluation! Maybe I have a new audience member ;) She's the person who originally hired me to work at BPL.

Examples are very important to me. I can make something up, sure thing, but it makes me very anxious. I am glad for the flexibility you (Prof. Thompson) have made possible. I also am very glad for the conversational tone the book's authors used. It helped a ton.

In appendix B I really "felt" the suggestions made in the Discussion section. I honestly was fired up for the teens! I wanted them to get that employee discount! I'm so proud of these (more than likely fictional) teens and how much they bought into the zoo.

I quite like the contract in appendix C. I recently keep being reminded that setting clear expectations at the start saves a world of hurt in the end. It seems like the budget is very thorough. I still can't get over how much money the book claims evaluators make in a year! I don't know that I hold much faith in focus groups as a data source, though. I feel like the one or two loudest set the answer for the entire group. I wonder if a simple post survey would give better results.

Week 5 Assignments

I am working on doing the Media Review Test

I moved the blog-style commenting to the "Journal" page.