Search Results

Keyword: ‘gradebook’

The CMS and the PLN

January 19th, 2010 jonmott Comments

It’s a been a long time since I blogged. Between sending my son off on a mission to Brazil, celebrating my 20th anniversary with my sweetheart, working on some offline writing projects, taking some time off for the holidays, getting back into the swing of things with the New Year and the new semester, and launching our loosely-coupled gradebook at BYU . . . Well, let’s just say I’ve been a little busy.

During my blogging hiatus, I did manage to get a paper published with my friend and colleague David Wiley. In the paper, we catalogue what we believe are the fundamental weaknesses of the CMS.

Writing this paper and taking some time away from blogging has allowed me to think some things through. As the title of my blog constantly reminds me, technology is only as good as the change and improvement it brings to teaching and learning. I have become supremely utilitarian when it comes to teaching and learning tools, applications and platforms. When it comes to the CMS and the Personal Learning Network (PLN), I readily admit that there are plusses and minuses to each. (For some thoughts ont the distinctions between PLNs and the PLE, see the references below).

I’m currently writing from ELI in Austin, Texas where I will make a presentation about open learning networks. As part of my preparation, I asked my PLN via Twitter (see below for a listing) for sources that delineate the strengths and weaknesses of the CMS and the PLN.

Here’s my meta-listing based on the research I did with David for our article, my own experience, and what I’ve gleaned from the resources shared by my online colleagues:

CMS Strengths

  • Simple, consistent, and structured
  • Integration with student information systems (SISs) so student rosters are automagically populated in courses
  • Private and secure (i.e., FERPA compliant)
  • Tight tool integration (e.g., quiz scores populated in gradebooks)
  • Supports sophisticated content structuring (e.g., sequencing, branching, and adaptive release)

CMS Weaknesses

  • As it is widely implemented, the CMS is time-bound (i.e., courses go away at the end of the semester)
  • Teacher, rather than student, centric
  • Courses are walled off from each other and from the wider Web, thereby negating the potential of the network effect
  • Limited opportunities for students to “own” and manage their learning experiences within and across courses
  • Rigid, non-modular tools
  • Interoperability challenges and difficulties (significant progress is being made on this front, but the ability to easily move data in and out of the CMS and to plug in alternative tools to replace or enhance native tools remains to be seen)

PLN Strengths

  • Almost limitless variety and functionality of tools
  • Customizable and adaptable
  • No artificial time boundaries–remains “on” before, during, and after matriculation
  • Open to interaction and connection with persons without regard to their official registration in programs or courses
  • Easily sharable with others both inside and outside of courses, programs, and institutions
  • Student-centric (i.e., each student selects and uses the tools that make sense for their particular needs and circumstances)
  • Compilable via simple technologies like RSS

PLN Weaknesses

  • Complex and difficult to create for inexperienced students and faculty members
  • Potential security and data exposure problems–FERPA issues abound
  • Limited institutional control over data
  • Absent or unenforceable SLAs–no ability to predict or resolve Web application performance issues, outages, or even disappearance

This is far from a comprehensive list, but it begins to clarify the picture in my mind. If we persist in an either-or debate about the CMS versus the PLN, we will be falling victim to what Jim Collins calls the “tyranny of or.” When faced with difficult decisions, we often cast them–artificially–as dichotomies. We must do this *or* that. Collins argues that the alternative is to find ways to leverage the “genius of and,” to bring together the best of both alternatives and create a chimerical best-of-both-worlds solution.

That is the vision of the open learning network–to bring together the best of the CMS and the best of the PLN to create a learning platform for higher education that meets the broad and diverse needs of faculty members and students engaged in the teaching and leaning process. Doing so is what I get paid to do–to provide technologies that will help teachers and learners be more effective without having to worry about technological complexities and navigating the swirling waters of apparently contradictory paradigms.

Please comment with your suggestions for improving my listing of strengths and weaknesses and I’ll edit the lists (with attribution). If you have additional resources to add, please share those as well.

More fun to follow soon . . .

_______________

RESOURCES

ELI’s “7 Things You Should Know About … Personal Learning Environments

Alec Couros: “What is a PLN? Or, PLE vs. PLN

Steve Wheeler: “It’s Personal: Learning Spaces, Learning Webs

David Hopkins: “Pedagogical Foundations for Personal Learning

John Seely Brown: “Minds on Fire: Open Education, the Long Tail, and Learning 2.0

Edublend: “Cloud Learning Environment – What is it?

grazadio_elearning’s PLE Bookmarks on Delicious

Things I’ve written on the subject …

Bush & Mott: “The Transformation of Learning with Technology: Learner-Centricity, Content and Tool Malleability, and Network Effects

Mott & Wiley: “Open for Learning: The CMS and the Open Learning Network

_______________________________________________________

Why Wave Won’t Replace the CMS

October 15th, 2009 jonmott Comments

Last week, Jeff Young added to the Wave hype with his frequently Tweeted and Re-Tweeted post “Could Google Wave Replace Course-Management Systems?

From what I’ve seen of Wave so far (I got my invite this week), I’d have to say, “No chance.” I have three reasons for making this conclusion:

1. Wave doesn’t replicate core CMS functionality.

The CMS has become the dominant technology of choice for instructors in higher education because it allows them to easily create course websites which are automagically populated with the students enrolled in their courses. The tools faculty members use within the CMS bear little if any resemblance to the feature set I’ve seen in Wave. Blackboard usage data at BYU indicates that faculty members overwhelmingly use the CMS for administrative purposes–content distribution, announcements, teacher-to-student communication, gradebooking, and (to a lesser extent) quizzing. This data is consistent with Morgan’s conclusion in her study of CMS usage in the University of Wisconsin System. The CMS, she observed, is “fundamentally a conservative technology … [for] managing groups, providing tools, and delivering content… Faculty use the CMS primarily as an administrative tool to facilitate quiz administration and other classroom tasks rather than as a tool anchored in pedagogy or cognitive science models” (1, 11).

While Wave appears to be a promising learning dialogue platform, it lacks the core CMS features that have driven faculty adoption. The same can be said of WordPress. When Blackboard melted down at CUNY earlier this year, many users turned to WordPress-powered blogs. But, as Blackboard CEO Michael Chasen observed, such Web 2.0 tools lacked key administrative applications like gradebooks.

Until alternative technology emerges that can blend new technologies like Wave with existing technologies like SIS-integrated gradebooks, the CMS is unlikely to lose its popularity with instructors.

2. Using Wave to teach would require embracing a completely different kind of learning paradigm.

The CMS is, indeed, a fundamentally conservative technology. Accordingly, it has been used primarily to support and reinforce the traditional lecture-driven, content-heavy, and instructor-centric model of instruction. Technologies like Wave, blogs, wikis, and Twitter are unlikely to replace the CMS anytime soon precisely because they are nothing like the CMS. They are all tools that facilitate dialogue around content and lack robust tools for content distribution. As such, they would require faculty members to focus less on making students “knowledgeable” (stuffing their heads with information) and more on making them “knowledge-able” (helping them effectively find and use information to solve particular problems).

Faculty members who remain focused on knowledge and content dissemination have little reason to move away from the CMS.

3. The CMS is unlikely to be replaced by any single technology–Wave or otherwise.

The CMS is the Swiss Army Knife of educational technology. It does lots of different things not very well. But it’s simple and convenient. For most institutions, replacing the CMS will require mimicking the ease of use and core functionality of the CMS. A hybrid of both “enterprise” technologies (e.g.,  student information systems, university managed secure gradebooks, quiz engines) and a mix of cloud-based, Web 2.0 applications (like Wave) is the next natural step in learning technology evolution. The advantage of such a technology ecosystem would be its malleability–it would be significantly more open, flexible, and interoperable than current CMS technology. Accordingly, it could simultaneously meet the needs of the majority of faculty members who still gravitate to the content-centric and administrative functions of the CMS while enabling those who are ready to move on to do so.

***

It is time to reframe the educational technology debate. In recent years educational technologists have been lining up on opposing sides of the PLN versus CMS debate. The reality, though, is that there are desirable features of both alternatives. It’s time to mash them up and create a chimerical new kind of technology that can give us the best of both worlds.

Is it a motorcycle? A car? It's both!

Is it a motorcycle? A car? It's both!

Assessment as a Social Activity

September 17th, 2009 jonmott Comments

I’ve been following the Washington State Harvesting Gradebook project for sometime and have been impressed with the intellectual rigor behind the project. That is clearly evidenced in this video overview of the project:

The comments at the end of the video are particularly significant. The narrator concludes:

At bottom, our research findings challenge many traditional assumptions. We are all understanding now that more than ever learning and assessment is or should be social. It benefits from the insight and diversity of broader engaged community. Expertise is distributed. It is not monolithic.

Assessment and assessment criteria, as Wiggins has long argued, should not be a secret. Students’ energy and creativity in their learning should not be held hostage by a classroom or coddled with promises of, “Someday in the real world …”

Intellectual capital is mistakenly recognized as a noun. Intellectual capital is a gerund. It is learning. And it is most valuable when it’s plural: “We learn.”

This emphasis on community and social connections in the learning process (which encompasses assessment) is reminiscent of Brown & Adler’s assertion that it is time for us move beyond the Cartesian premise of “I think, therefore I am” and embrace the more realistic and rich notion of “we participate, therefore we are.” As we make learning and learning assessment more social, public, and transparent, learners will be naturally more invested and engaged in the learning process because they become co-creators and co-custodians of the experience. The WSU model of social assessment that incorporates external assessment of student work extends this openness into the broader community in which our students will work and continue to learn after graduation.

As we move forward with the development of the BYU gradebook, we will incorporate these elements of social assessment, facilitating self-assessment, peer assessment, instructor assessment, and external (third-party) assessment. The resulting treasure trove of authentic assessment data will more than meet our accreditation needs (so long as our program outcomes, learning activities and assessments are aligned!). But far more importantly, this rich, community-centric assessment approach will deepen and enrich the student learning experience.

Deja Vu All Over Again – Blackboard Still Stuck in the Innovator’s Dilemma

July 24th, 2009 jonmott Comments

It’s been a week (or so) since BbWorld. I’ve had the chance now to ruminate about what I saw and heard there. I wanted to let things rattle around in my head a bit before passing judgment on Blackboard’s message this year because it was increasingly clear to me that my assessment of BbWorld 2009 would be virtually the same as my assessment from 2008–Blackboard is improving at the margins, but is not addressing the fundamental weaknesses of the CMS (I use this term generically, lumping Blackboard together with Moodle, Sakai, D2L, etc.).

Improvement at the Margins

Given that BbWorld was in Washington, D.C. this year, it seems appropriate to use a political analogy here. In 1990, George C. Edwards III published At the Margins: Presidential Leadership of Congress. In this now classic study of the president’s ability to lead Congress and enact significant policy change, Edwards concluded that the presidency is so hemmed in by countervailing pressures and influences (in the form of 535 members of Congress, cabinet members, bureaucrats, interest groups, voters, the media, etc.) that presidents should not be expected, save under dramatic and rare circumstances, to effect significant change. Rather, Edwards concludes, presidents have historically been most effective when they have sought to influence change “at the margins,” moving policy incrementally in their preferred direction.

Blackboard, much like the President of the United States, is hemmed in by a large client base (variously represented by university administrators, IT staff, faculty, and students), competing visions within the company, a board of directors and stockholders, etc. It shouldn’t be surprising to anyone, then, that Blackboard’s innovations and improvements are “at the margins.” It’s exceedingly rare for a company with Blackboard’s inertia to make dramatic, revolutionary changes in the products they offer.

BbWorld Announcements & Observations

So what kinds of innovations and improvements did Blackboard announce at BbWorld 2009? I catalogued and reacted to what I heard via Twitter during the conference. For what it’s worth, I begin with a selection of my tweets, first from the keynote (I’ve opted to keep these in reverse-chronological order, as is the convention for a Twitter feed):

Then from the Listening Session:

Deja vu All Over Again

While I’m very encouraged by Blackboard’s announcement (re-iteration) of it’s intent to fully support the Common Cartridge standard and to move toward opening up its database schema for administrators, I’m left wanting much, much more. All of the major announcements–the partnership with Echo360, the acquisition of Terribly Clever, the integration with Wimba (not really new news), the Kindle integration, the focus on closing tickets faster–had little to do with the core concerns about the CMS I have been blogging about for the last year. All of Blackboard’s announcements were about improvements at the margin.

As a checkpoint, I went back and re-read my post from one year ago, written just after BbWorld 2008. Forgive me for regurgitating large portions of it here, but it’s striking how similar my reactions to this year’s BbWorld are to those of last year:

As Clayton Christensen has famously observed, the producers of innovative products gradually lose their creative, innovative edge as they acquire and then seek to protect market share. When a company’s innovations result in significant profits, managers generally find themselves face to face with the innovator’s dilemma. To remain successful, Christensen argues that companies need to listen “responsively to their customers and [invest] aggressively in the technology, products, and manufacturing capabilities that [satisfy] their customers’ next-generation needs.” However, these very same behaviors can create blind spots for innovators. By simply providing incremental improvements to existing products, companies run the risk of missing major, paradigm-shifting innovations in their market spaces. Likewise, they’re in danger of focusing too much on their existing customer bases instead of new potential customers who currently don’t user their products (non-consumers). These twin dangers leave erstwhile market leaders susceptible to disruptive technologies, provided by firms who aren’t stuck in current paradigms or too narrowly focused on pre-defined customer segments.

Blackboard finds itself squarely in the midst of this classic problem. They have a large and fairly stable customer base. Incremental feature enhancements, improved customer service and product stability are likely to keep most of their customers satisfied for time being. But what of the disrupters in the market place? If one considers open source CMS alternatives like Sakai and Moodle to be the most-disruptive players in the market, Blackboard’s strategies appear to be on the right track.

Perhaps not surprisingly (I suppose I predicted this), Blackboard did this year exactly what they did last year and exactly what Christensen cites as the pattern of incumbent market leaders: they announced new feature enhancements and indicated continued attention to and investment in customer service and product stability. As I wrote one year ago:

While I applaud these innovations as good steps in the right direction, there remain fundamental flaws with Blackboard’s (and virtually every other CMS provider’s) underlying infrastructure. For all of the new window dressing, Blackboard remains first and foremost a semester-based, content-delivery oriented, course management system. The software is not (at least noticeably) evolving to become a student-centered learning management system. And while the addition of wikis and blogs inside the Blackboard system is as welcome improvement, there is still little or no integration between student learning tools “inside the moat” and outside of it “in the cloud.”

It is for these reasons that I don’t count Sakai, Moodle, D2L or Angel [which Bb acquired since BbWorld 2008] amongst the biggest, long-term threats to Blackboard. Disruption will, I believe, come from another direction.

From whence will disruption come? More from my post last year:

In Christensen’s newest book, Disrupting Class, he and his co-authors argue that the real disruption in educational technology will come (and is already coming) via learner-centered technologies and networking tools. A rapidly growing number of people are creating their own personal learning environments with tools freely available to them, without the benefit of a CMS. As Christensen would say, they have hired different technologies to do the job of a CMS for them. But the technologies they’re hiring are more flexible, accessible and learner-centered than today’s CMSs. This is not to say that CMSs are about to disappear. Students enrolled in institutions of higher learning will certainly continue to participate in CMS-delivered course sites, but since these do not generally persist over time, the really valuable learning technologies will increasily be in the cloud.

Open learning networks have the potential to bring together the world of the CMS (or better yet “institutional learning networks”) and the world of PLEs together. The next big challenge ahead of us is to figure out ways to create autonomous, institution-independent “learner spaces” that provide home bases for learners that can bridge the two worlds. In these spaces, learners would ideally aggregate relationships, artifacts, and content from ALL of their learning activities, be they digital or analog, online or offline, synchronous or asynchronous, from one institution or many.

I heard virtually nothing at BbWorld this year which would suggest that Blackboard is actively engaged in adapting and evolving to address this challenge. Rather, they continue to innovate at the margins, maintaining their status quo, market-leading position.

What’s the Broken?

@z_rose recently blogged that the problem with the “one-stop-shop” Virtual Learning Environment (VLE) (another frequently-used term for the CMS) is that it is aimed at both learning administration and learning facilitation. These are not, she astutely notes, the same things and VLEs end up doing both poorly.

Both administration and pedagogy are necessary in schools. They are also completely different in what infrastructure they require. This (in my opinion) has been the great failing of VLEs – they all try to squeeze the round pedagogy peg into the square administration hole.

It hasn’t worked very well. Trying to coax collaboration in what is effectively an administrative environment, without the porous walls that social media thrives on, hasn’t worked. The ‘walled garden’ of the VLE is just not as fertile as the juicy jungle outside, and not enough seeds blow in on the wind.

That’s why I’m always cautious of the ‘one-stop-shop’ approach in education – administration and pedagogy are very, very different shops. It’s like having a fishmongers and a haberdashers sharing the same store – no discernible upsides, but a LOT of downsides (stinky fabric springs to mind).

Blackboard and every other CMS / VLE have become exceedingly efficient course content and course administrivia management tools. If data from BYU’s Blackboard usage surveys can be taken as a reasonable guide, most faculty members use Blackboard for administrative, not teaching and learing, purposes, i.e., content dissemination, announcements, e-mail, and gradebooking (70% plus use Bb for these purposes). Dramatically smaller portions (less than 30%) use the teaching and learning tools provided Blackboard (e.g., quizzes, discussion boards, groups, etc.). Increasingly, they’re going to the cloud to use tools that are far better and more flexible than those provided natively inside the CMS.

In 2004, George Siemens wrote, “The real issue is that LMS vendors are attempting to position their tools as the center-point for elearning – removing control from the system’s end-users: instructors and learners.” This is still all-t0o-true five years later. In most CMS implementations, it is exceedingly difficult (if not impossible) for teachers and especially individual learners to take control of the learning environment and shape it to their particular needs. For example, by default, students are generally not able to start their own discussion threads in CMS-delivered courses. Siemens elaborated on these end-user roadblocks, noting that LMSs roundly fail in three significant ways:

  • The rigidity and underlying design of the tool “drives/dictates the nature of interaction (instructors-learner, learner-learner, learner-content).”
  • The interface is too focused on “What do the designers/administrators want/need to do?” rather than on “What does the end user want/need to do?”
  • “Large, centralized, mono-culture tools limit options. Diversity in tools and choices are vital to learners and learning ecology.”

Notably, the absence of LMS-integrated synchronous conferencing and collaboration tools has been largely remedied within the various CMSs. But these other three substantial shortcomings have yet to be addressed. And I would add a critically important fourth weakness–today’s CMSs do not support continuous, cumulative learning throughout a student’s career at an institution, let alone throughout their life after they exit our institutions. As I have written previously, students are completely at the mercy of the institution when it comes to their “presence” and participation in a CMS. They are placed in arbitrarily organized sections of courses for 15-week periods and then “deleted,” as if they never existed in the system. As David Wiley has pondered, how many of us would use Facebook if Facebook deleted our friend connections and pictures every four months?

The fundamental dilemma with the CMS as we know it today is that it is largely a course-centric, lecture-model reinforcing technology with its center of gravity in institutional efficiency and convenience. As such, it is a technology that inclines instructors and students to “automate the past,” replicating previous practice using new, more efficient and more expensive tools instead of innovating around what really matters–authentic teaching, learning, and assessment behaviors.

Blackboard’s Opportunity?

Lest I come across as a Blackboard/CMS naysayer or doomsayer, I should note that, in their early days, Blackboard and other CMSs were the disruptive technology. They were the source of innovation and new thinking about how we organize to teach and learn. However, roughly a decade after the inception of the CMS, the academic community finds itself again ripe for disruption, not only of the technology we use to “manage courses” but in the very system itself. While many will continue to innovate at the margins, there are large crowds of non-consumers out there clamoring for something that meets their needs. At BYU, for example, 25% of our faculty members opt to use a blog, a wiki, or a custom-built course website instead of Blackboard or another CMS. These are the non-consumers Christensen reminds us that we need to figure out how to serve. The same goes for the “non-traditional” students who either aren’t wired to learn the way we’re organized to teach them (in course-sized chunks, bundled in units of time we call semesters) or who, for a variety of reasons, don’t have access to our institutions.

Blackboard still has the opportunity to facilitate discussion and innovation around these critical issues. The company took an important step in this direction by organizing the “Pipeline Matters” session the day before BbWorld, bringing together educational leaders from K12, community colleges, traditional “higher ed” instituitons, and edudcational associations. As a fortunate participation in this conversation, I recognize and appreciate Blackboard’s unique position (with its roughly 3000 client institutions) in the educational space to bring together a broad and diverse set of educational players to address issues such as the one we did last week–how can we improve our efforts to keep students in school and to help them easily re-enter and succeed when they, despite our best efforts, leak out of the “pipeline”?

These sorts of questions are critically important not only to educators, but also to Blackboard’s immediate future and direction because they can compel the company to get outside its comfort zone and rethink how it does what it does and why it does it.

Blackboard can still play a leading role in education. But it needs to think more about end-users and about non-consumers, not just about the universtity administrators who purchase and implement their products. That’s an admittedly tall order for a publicly-traded corporation to take on. But, as Christensen argues, they have to figure out a way to do so if they’re to remain relevant. That’s precisely the innovator’s dilemma.

As I conclude last year, if Blackboard doesn’t innovate, someone else will.

And it won’t be long.

Reliability, Validity and Loosely-Coupled Assessment

June 25th, 2009 jonmott Comments


Last week Jeremy Brown wrote a thoughtful response to my post about “PLNs, Portfolios, and a Loosely-Coupled Gradebook.” Jeremy expressed concern that my notion of “loosely-coupled” assessment doesn’t adequately address the issues of validity and reliability. He also issued a “warning” about the “assessment minefield into which” I am marching.

While fully appreciating Jeremy’s evaluation bona fides, my working definitions of “reliability” and “validity” are slightly more straightforward (and conventional?) than those he uses. Simply put, I take validity to mean the accuracy with which I am measuring a variable. In layman’s terms, I ask, “Am I actually measuring what I think I’m measuring?” Reliability, on the other hand, refers to the consistency of those measurements: “Am I measuring the same variable consistently over time (at various points of observation) and across multiple subjects?” The old bathroom scale example has always helped me keep these two concepts straight. If a person weighs 200 pounds and the bathroom scale says they weigh 200 pounds, then the measurement is valid. However, if the scale indicates different weights each time a person steps on it (even though their weight hasn’t changed), the measurement isn’t reliable. On the other hand, if the scale consistently indicates that a 200 pound person weighs only 150 pounds every time they weigh themselves, the measure is reliable (consistent) but not valid (accurate).

It’s important to be clear about what we mean by validity and reliability because Jeremy’s central concern seems to be that greater student ownership of and responsibility for portfolios will degrade the “reliability of the judgments passed” on individual student work. He posits two reasons that this would be the case:

  1. The “degree of difficulty” versus “relative facility” of the work completed and submitted via a portfolio.
  2. Selection of artifacts-A student might inadvertently include artifacts which under-represent her or his actual expertise or skill level.

I concur that the variability in student facility with various digital technologies might result in unreliable measures of student ability and skill. While part of being digitally literate in 2009 means being able to create and publish content online at some minimum level of professional acceptability, evaluators should be careful not conflate portfolio design prowess with content area expertise. The same is true when evaluators read papers-students are expected to write at a basic minimum level of professional acceptability, but eloquent prose is not the same as subject matter achievement. Consequently, it is critically important that those who require portfolios to be abundantly clear about the purposes of the portfolio “assignment” and how the portfolio will be assessed. Portfolios should then be assessed accordingly.

When thinking about the purposes of portfolios, administrators and faculty members should be careful to distinguish between the various goals they might have for student portfolio creation and evaluation. On the one hand, portfolios might be encouraged or assigned to help students reflect on their learning, to engage with others about what they’ve learned and how they’ve learned it, to present their work to various audiences, and to develop essential digital literacy and communication skills. When pursuing such goals, validity and reliability concerns are secondary to the process of creating and maintaining a portfolio, so highly student-centric, student-owned and operated portfolios are desirable.

On the other hand, if the purpose of a portfolio is to provide a consistent, aggregated view of a students’ performance through their time at an institution or in an academic program, reliability and validity are central concerns and the cautions Jeremy offers are more immediately relevant. It is my belief, however, that we should strenuously avoid assigning portfolios for purely institutional or program assessment purposes. If our programs aren’t designed in such a way that examples of student work (whether compiled into portfolios or not) is assessed (and possibly collected and aggregated) along the way, then it seems appropriate to redesign the programs instead of bolting an artificial evidence gathering requirement on at the back end.

Once again, we need to begin with the end in mind. What is it we want our students to  become? What experiences do they need to become such? What artifacts are the natural result of (or natural extensions) of these experiences? How will we consistently evaluate these artifacts to give individual students feedback about their performance and growth? How will we aggregate these evaluations to determine our institutional or program level performance? These are the questions that should drive our portfolio and assessment strategies-not external accreditation requirements. If we focus on these student-centric questions, meeting even the most stringent accreditation requirements will be a relatively simple afterthought.

PLNs, Portfolios, and a Loosely-Coupled Gradebook

June 16th, 2009 jonmott Comments

Note: In this post, I reference articles from the Winter 2009 edition of Peer Review, the theme of which was “Assessing Learning Outcomes.” I highly recommend the entire issue if you’re interested in student learning assessment and portfolios. I recently provided an overview of BYU’s loosely-coupled gradebook strategy at TTIX 2009. (We are currently building a standalone gradebook in partnership with Orem, Utah based startup Agilix.) As part of my presentation at TTIX, I also described our plans to leverage the same technology we’re building into the gradebook to implement a loosely-coupled portfolio assessment tool.

The driving purpose behind these efforts is to bridge the gap between our institutional network and the cloud, between the predominant “course management system” (CMS) paradigm and the emerging model of personal learning networks (PLNs), between student-centered and institution-centered portfolios, etc. I maintain that bridging this gap is a necessary condition for the significant transformation of learning via technology. Until learning tools and content become more malleable (i.e. open, modular, and interoperable) we will not realize the full potential of an interconnected, networked world in education.

Student Learning Portfolios & Institutional Assessment

Portfolios are increasingly at the nexus of student learning, institutional assessment, PLNs, CMSs, and assorted other aspects of the higher ed milieu. Student learning portfolios are essential in the movement toward more valid and authentic assessment in higher education. However, the focus on institutional and program assessment has, at least in some instances, diverted our attention from our primary objective of improving student learning. As Trent Batson observed in 2007, that the initial effort to enhance student learning with portfolios has been “hijacked by the need for accountability” to boards of education and accrediting bodies.

This trend is worrisome. If the focus on portfolios shifts primarily to institutional and program assessment, we will have missed out on the essential value of portfolios. Portfolios have the potential to galvanize and enhance student learning. As Miller and Morgaine have observed: “E-portfolios provide a rich resource for both students and faculty to learn about achievement of important outcomes over time, make connections among disparate parts of the curriculum, gain insights leading to improvement, and develop identities as learners or as facilitators of learning.”

Given the potential benefits of portfolios, I believe that student learning portfolios should, first and foremost, belong to and be maintained by individual learners. Gary Brown supports this view, maintaining that portfolios should be student (and not institutionally) operated: “A real student-centered model would put the authority, or ownership, of [ePortfolios] in the hands of the students: They could share evidence of their learning for review with peers, and offer that evidence to instructors for grading and credentialing.” Such an approach increases student ownership and responsibility for learning. It also affords portability and longevity since students are not dependent upon a particular institution (or set of institutions) to provide them with portfolio technology and storage space.

Clark & Enyon have argued that have to get past this tension between student and institutional portfolios: “The e-portfolio movement must find ways to address [institutional assessment] needs without sacrificing its focus on student engagement, student ownership, and enriched student learning.”

Bridging the Gap

So exactly how can we bridge the gap between student-centered learning portfolios and institutional assessment needs? I propose that the loosely-coupled gradebook strategy we’re pursuing can be leveraged to provide a viable solution to this problem.

Here are the dimensions of a loosely-coupled portfolio assessment strategy:

  1. Institutions of higher learning should focus on what they do best and on what only they can do. Namely, they should admit and register students, manage course enrollments and  degree program rosters, and maintain secure records and communications tools for faculty and students engaged in the learning process.
  2. We can then leverage the best online, third-party applications for student publishing, networking, and portfolio creation. Individual institutions (or even institutions working together) would be hard pressed to produce applications comparable in quality and stability to Google Docs, YouTube, Blogger, Acrobat Online, MS Office Live, Wikispaces, and WordPress.
  3. Teachers and learners should embrace the power of the network to enhance, extend and improve learning. Even if institutions could develop and deploy better tools than those freely available online, it would be a bad idea to do so. The fundamental value proposition of these apps is that, since they live in the cloud, they’re accessible anytime, anywhere, by anyone. The openness this affords promotes greater transparency and expanded opportunities for collaboration.
  4. Students should be encouraged to be effective, technologically literate, digital citizens who are proficient using a variety of online tools. As they participate in the learning process, they should regularly save and aggregate their work, packaging and repackaging it for various audiences. One of these audiences might be those responsible for degree program review and assessment.
  5. Once students have assembled their collections of learning artifacts, metacognitive commentary, and portfolios, they might then be required to simply “register” the URLs of their portfolios and artifacts with their institution. Those conducting program and institutional assessment would then use a lightweight “overlay” tool to review and assess submitted student work.

The various aspects of this approach and the associated work flow might look something like this: Portfolio Assessment Diagram

The Power of a Loosely-Coupled Strategy

The “open learning network” (OLN) strategy I’ve written about from time to time is based on several value propositions. One is that institutions should do what they do best (manage student data, facilitate secure communication between teachers and learners) while leveraging third-party, cloud-based applications for such things as personal publishing and collaboration. The loosely-coupled gradebook strategy described in previous posts is a key component of this larger idea.

Another central value proposition of the OLN is the connectedness it facilitates between teachers and learners both within and without institutional boundaries. When learners not only consume online content, but also refine, improve, remix, mashup, and create new content themselves during the learning process, their learning is more authentic, meaningful, and enduring. And they build deeper, more profound connections between facts, concepts, and the other human beings they interact with. This notion of of connectedness is a fundamental aspect of what we consider education and literacy today.

George Siemens blogged today that:

“Not only are we socially connected in our learning, but the concepts that form our understanding of a subject also reveal network attributes. Understanding is a certain constellation (pattern) of connections between concepts. . . . being a literate person is not so much about what you know, but about how you know things are connected.”

I concur. As we continue to pursue the OLN vision, it is essential that we facilitate opennes in the learning process to promote greater interaction and connections with content, people, cultures, and places. It is with this end in mind that we ought to promote personal learning networks (PLNs), OLNs, and loosely-coupled gradebooks. We want our students to be literate, connected, and efficacious life-long learners who make their homes, their communities, their workplaces and the world better places than they found them.

Loosely Coupled Gradebook Presentation @ TTIX 2009

The slides for my presentation at TTIX 2009 about BYU’s loosely coupled gradebook project.

And a link to the UStream capture of the presentation.

An Open (Institutional) Learning Network

April 9th, 2009 jonmott Comments

I’ve been noodling on the architecture of an open learning network for some time now. I’m making a presentation to my boss today on the subject and I think I have something worth sharing. (Nothing like a high-profile presentation to force some clarity of thought.)

I wrote a post last year exploring the spider-starfish tension between Personal Learning Environments and institutionally run CMSs. This is a fundamental challenge that institutions of higher learning need to resolve. On the one hand, we should promote open, flexible, learner-centric activities and tools that support them. On the other hand, legal, ethical and business constraints prevent us from opening up student information systems, online assessment tools, and online gradebooks. These tools have to be secure and, at least from a data management and integration perspective, proprietary.

So what would an open learning network look like if facilitated and orchestrated by an institution? Is it possible to create a hybrid spider-starfish learning environment for faculty and students?

The diagram below is my effort to conceptualize an “open (institutional) learning network.”

Open Learning Network 2.0

There are components of an open learning network that can and should live in the cloud:

  • Personal publishing tools (blogs, personal websites, wikis)
  • Social networking apps
  • Open content
  • Student generated content

Some tools might straddle the boundary between the institution and the cloud, e.g. portfolios, collaboration tools and websites with course & learning activity content.

Other tools and data belong squarely within the university network:

  • Student Information Systems
  • Secure assessment tools (e.g., online quiz & test applications)
  • Institutional gradebook (for secure communication about scores, grades & feedback)
  • Licensed and or proprietary institutional content

An additional piece I’ve added to the framework within the university network is a “student identity repository.” Virtually every institution has a database of students with contact information, class standing, major, grades, etc. To facilitate the relationships between students and teachers, students and students, and students and content, universities need to provide students the ability to input additional information about themselves into the institutional repository, such as:

  • URLs & RSS feeds for anything and everything the student wants to share with the learning community
  • Social networking usernames (probably on an opt-in basis)
  • Portfolio URLs (particularly to simplify program assessment activities)
  • Assignment & artifact links (provided and used most frequently via the gradebook interface)

Integrating these technologies assumes:

  • Web services compatibility to exchange data between systems and easily redisplay content as is or mashed-up via alternate interfaces
  • RSS everywhere to aggregate content in a variety of places

As noted in previous posts, we’re in the process of building a stand-alone gradebook app that is consistent with this framework. We’re in the process of deciding which tools come next and whether we build them or leverage cloud apps. After a related and thought-provoking conversation with Andy Gibbons today, I’m also contemplating the “learning conversation” layer of the OLN and how it should be achitected, orchestrated and presented to teachers and learners . . .

While there’s still a lot of work to do, this feels like we’re getting closer to something real and doable. Thoughts?

Loosely Coupled Gradebook Specs

January 30th, 2009 jonmott Comments

We’ve been making good progress on our loosely coupled gradebook project here at BYU. I’m posting a link to our working specifications document for review and comment.

The broad purposes of the project are to develop a web-based gradebook that will provide:

  • Integration our course management system(s)
  • Integration with our high-stakes proctored testing environment
  • Multiple “on-ramps” (integration points with other tools, systems and web 2.0 apps) for faculty to input and retrieve student performance measures
  • Easy importing and exporting of data
  • Flexible grade calculation
  • Direct interface with our student information system for grade posting

This is a key component of our larger open learning network strategy.

Feel free to review our specifications document and let us know what you think.

Tool & Content Malleability

January 28th, 2009 jonmott Comments

I’ve recently finished an article with Mike Bush (a colleague here at BYU) in which we coin what we believe to be a new term in the standards debate–”content & tool malleability.” Our piece is modestly titled “The Transformation of Learning with Technology: Learner-Centricity, Content and Tool Malleability, and Network Effects” and will appear in the March-April edition of Educational Technology Magazine. A couple of months after it’s published, I’ll be able to publish the article in its entirety here. For now, I want to provide a preview of our notion of malleability. We suggest that malleability has three key attributes: openness, modularity, and interoperability and that teaching and learning tools and content must become more malleable if they are to become authentically reusable, remixable, redistributable, repurposable, etc.

We cite IBM’s ultimately successful implementation of the principles of modularity and interoperability which enabled the PC-maker to call on outside vendors for parts for their machines, creating an essentially malleable computing platform:

Their rejection of proprietary technology in favor of openness created the opportunity for IBM to call on Microsoft to develop the operating system and for a host of other companies (including Microsoft!) to go on to create thousands upon thousands of software applications, guaranteeing he long-term success of IBM’s initial design. Furthermore, competing companies that chose a proprietary and closed approach for their hardware, software, or both, (e.g., Texas Instruments, Amiga, Atari, Commodore, and Radio Shack) are nowhere to be found among Twenty-First Century personal computers. Even Apple, with the initial version of their innovative Macintosh, came close to meeting disaster until they opened things up with their Macintosh II (Bush, 1996). In the end, the nature of IBM’s approach not only ensured success in their initial venture, but the continued application of the same principles over the years by IBM’s successors also makes it possible for today’s machines to run much of the same software that was created for the original IBM PC.

Among the principles of openness, modularity, and interoperability that brought success to the IBM-PC venture, the importance of modularity seems perhaps preeminent and has been documented in detail by scholars at the Harvard Business School (Baldwin & Clark, 2000). In their initial work, they analyzed how modularity evolved as a set of design principles during the period between 1944 and 1960. Then using Holland’s theory of complex adaptive systems as a theoretical foundation, they explain how the design principles they identified went on to radically transform the information technology industry from the 1960s through the end of the century. They show how modular design and design processes have fostered change in the industry as it moved from one consisting of a few dozen companies and dominated by IBM to one that involves over a thousand companies and in which IBM plays a significantly lesser role. For example, the “packaged software” sector in the information technology industry consisted of about seven firms in 1970 that were valued at just over $1 billion (as measured in constant 2002 dollars). Thirty-two years later that sector had grown to 408 companies with a market capitalization of $490 billion (Baldwin & Clark, 2006).

Unfortunately, the application of the principles that made such developments possible in the computer industry is rare to nonexistent in many areas of education today. The education   technology landscape is best characterized by monolithic, enterprise technology silos with rigid, often impenetrable walls. Course management systems (CMSs), for example, are generally “all-or-nothing” propositions for institutions, teachers, and students. That is, even if you use an open source CMS like Moodle, you are (without significant customization) bound to use Moodle’s content publishing tool, Moodle’s quiz tool, Moodle’s gradebook, etc. Moreover, the CMS paradigm itself, tied as it is to semester calendars and time-bounded learning experiences (courses), severely limits learning continuity and persistence. Teachers and students are not free to choose the right / best / preferred tool for each teaching or learning activity they undertake, thus creating a technology paradigm that artificially limits possibilities and forecloses optimal teaching and learning choices.

The monolithic and rigid nature of today’s learning tools and content mirrors the way content has traditionally been made available to faculty and students—books and other resources (including online courses) have generally been all-or-nothing, take-them-or-leave them propositions. A similar business model was prevalent in pre-Internet days, resulting in CD-ROM databases that were more expensive than many potential consumers could afford. One analysis compared this marketing approach to a public water distribution system that would require selling the whole reservoir to each household rather than placing a meter at individual homes.

New approaches to content distribution, however, particularly the OpenCourseWare (OCW) and Open Educational Resource (OER) movements, promise to make a vast array of content open to instructors and students to reuse, revise, remix, and redistribute. The OCW Consortium, beginning with MIT in 2002, has now grown to include hundreds of institutions around the world that have chosen to place course materials online.iv The efforts of these institutions have spawned a related effort, dubbed Open Educational Resources (OER), to make learning materials and content (as opposed to complete courses) freely available as well (Breck, 2007). Around the world, millions of people, inside and outside of academia, are publishing content under Creative Commons licensing, making that content open for others to use in a variety of ways. We are rapidly approaching the tipping point at which a critical mass of participants in open content and open learning is sufficient to exponentially increase the value of each additional participant in the network (as described in the next section).

The stunning reality of the new standard of openness is that it is quite simple. The key is to create lots and lots of open content and provide open, easy access to it. While technical standards and specifications, such as the Shareable Content Object Reference Model (SCORM), are important when it comes to producing indexing, discovering, sequencing, packaging, and tracking of content, openness by itself is a paradigm shifting approach in the teaching and learning world.

The fact that content is openly available and usable is just as important as any particular technical feature of that content. While openness stands by itself as a radical new innovation, we need to avoid the temptation to downplay the importance of standards and specifications, for they are essential to the realization of the vision of open, modular, and interoperable learning environments.

This reality is not without historical precedent. Printing became affordable and available in large part due to what we today call standards. Indeed, as one scholar declared, “This then—the standardization and rapid multiplication of texts—was what the Fifteenth Century invention of printing made possible” (Bühler, 1952). Bühler also pointed out that printing’s contributions went beyond the replication issue, stating that modern scholarship only became possible with the production of identical copies of texts. Although the value of mass duplication is not to be discounted, the fact that scholars could reference each other’s work represented enormous value. Given this standardization, they were thus able to criticize, comment upon, connect to, and build upon what had come before. In many ways, printing standards facilitated the first widespread appearance of mashups in human history. The existence of identical copies was but one characteristic that facilitated the eventual widespread availability of books. In addition, several other factors contributed to the production process itself, eventually increasing the opportunity for wider distribution.

Although SCORM is not perfect, it at least began to address the issue of establishing a framework within which learning content can be made to interoperate in a variety of settings. Just as SIF opens up the opportunity for reuse of information created and used by various operational elements of schools, SCORM still holds the promise to facilitate the sharing of learning content, not only across learning management systems but also across tools that facilitate the design and development of learning content. In addition, common authentication schemes (e.g., OpenID) built upon Web services interoperability will ultimately allow learners to seamlessly navigate multiple Web-based teaching and learning applications, opening up possibilities for personal learning environments in which multiple sources of content and experiences work together to help students learn in ways that are tailored to each individual.

With developments like SCORM 2.0 on the horizon, as well as increasingly powerful software, hardware, and networking tools, technological barriers are falling. The challenge now is to harness these new enabling technologies to create more open, modular, and interoperable learning content as well as production and learning tools that are each malleable with respect to their individual functionality. Together, these technologies will help further the transformation of education from a teaching-oriented enterprise to a learning-centered one.

As noted, the entire piece will be available soon. We hope this notion of malleability helps move the conversation forward. What are your thoughts? Are we on the right track?

References:

Baldwin, C. Y., & Clark, K. B. (2000). Design rules,volume 1: The power of modularity. Cambridge, MA: MIT Press.

Baldwin, C. Y., & Clark, K. B. (2006). Modularity in the design of complex engineering systems. In D. Braha, A. A. Minai, & Y. Bar-Yam (Eds.), Complex engineered systems: Science meets technology (pp. 175–205). New York: Springer.

Breck, J. (2007, Nov./Dec.). Introduction to special issue on opening educational resources. Educational Technology,
47(6), 3–5.

Bühler, C. F. (1952). Fifteenth century books and the twentieth century. New York: The Golier Club.

Bush, M. D. (1996, Nov.). Fear & loathing in cyberspace: Of heroes and villains in the information age. Multimedia
Monitor; http://arclite.byu.edu/digital/heroesa5.html.