BART Perks Phase II Evaluation Report

BART Perks Phase II

Evaluation Report

San Francisco Bay Area Rapid Transit District

September 2019

 
 
i 
TABLE OF CONTENTS 
Page 
Acknowledgements ................................................................................................................................. iv 
Executive Summary .................................................................................................................................. 1 
Introduction .............................................................................................................................................. 8 
Chapter 1: Experimental Design and Recruitment ................................................................................. 12 
Chapter 2: System Design and Incentive approach ................................................................................ 16 
Chapter 3: Response to Incentives ......................................................................................................... 23 
Chapter 4: Participant Characteristics and Feedback ............................................................................. 34 
Chapter 5: Cost Effectiveness and Cost to Scale .................................................................................... 41 
 
   

 
 
ii 
LIST OF TABLES  
Page 
Table ES. 1: Change in Share of Participant Trips During Incentivized Periods - Average Shift ............ 3 
 
Table 1: Extra Reward Offers ................................................................................................................. 20 
Table 2: Share of Users by Reward Value (Cumulative Over the Program)......................................... 23 
Table 3: Share of Rewards by Reward Type .......................................................................................... 24 
Table 4: Change in Share of Participant Trips During Incentivized Periods - Average Shift ................ 26 
Table 5: Comparison of Behavior Change Between Prior Perks Participants and New Recruits ........ 29 
Table 6: Which of the Following Offers to Earn Points Do You Remember Seeing? ........................... 30 
Table 7: Extra Reward Offers (March 2019 – June 2019) ..................................................................... 31 
Table 8: Participant Response to Extra Reward Offers ........................................................................ 32 
Table 9: Share of Transbay Trips and Commuters by Perks I and Perks II ........................................... 34 
Table 10: Percentage of Participants by Barriers to Following Shift Commute Offers ....................... 39 
Table 11: Barriers to Shifting Commute Time Earlier and Later........................................................... 40 
Table 12: Potential Grant Funding Sources for BART Perks ................................................................. 43 
 
LIST OF FIGURES 
Page 
Figure ES. 1:  Shift Commute Offer Example ....................................................................................... 2 
Figure ES. 2: Share of Participant Travel by Time of Day, Before and During the Perks Program, by 
Type of Offer ............................................................................................................................................ 4 
 
Figure 1: BART System Map Highlighting the Transbay Corridor ........................................................... 8 
Figure 2: Recruitment Flyer (Front and Back) ....................................................................................... 14 
Figure 3: Travel Behavior of Early and Delayed Offer Groups Prior to Program ................................. 15 
Figure 3: Structure of Perks Software Systems ..................................................................................... 17 
Figure 4: Shift Commute Offer Example (Page 1 and 2) ....................................................................... 19 
Figure 5: App Redemption Page ............................................................................................................ 22 
Figure 6: Share of Redeemed Value by Type of Card ........................................................................... 25 
Figure 7: Share of Participant Travel by Time of Day, Before and During the Perks Program, by Type 
of Offer ................................................................................................................................................... 27 
Figure 8: Inbound A.M. Transbay Participant Trips Before and During BART Perks Phase I Program 28 

iii

Figure 9: Comparison of Demographic Characteristics of Perks Phase I & II participants and all BART

riders ....................................................................................................................................................... 35

Figure 10: Average Cumulative Points Earned Compared to Satisfaction with the Perks II Program 36

Acknowledgements

Funding for BART Perks Phase II was generously provided through the Federal Transit

Administration’s Pilot Program for Transit Oriented Development, which encourages economic and

ridership development in major transit corridors. The grant (CA-2016-066-00) funded two sub-

projects (BART Perks and the Downtown Oakland Specific Plan), both focused on BART’s Transbay

Corridor connecting downtown Oakland and San Francisco.

A team of dedicated consultants supported the Perks II project. Metropia developed the incentives

software platform, incentives algorithm, and crowding predictive model under the direction of Drs.

Yi-Chang Chiu and Ali Arian. The Behaviouralist, led by Dr. Rob Metcalfe, developed the experimental

design and program evaluation. TransSight, under the direction of Satinder Bhalla, led the front-end

software development and engineering.

Multiple staff from several BART departments led or supported components of the work. BART’s

Office of the Chief Information Officer, under Angie West, oversaw the software development and

systems integration. BART’s Marketing Department, under Aaron Weinstein, supported the

experimental design, project evaluation, user experience research, and user surveys. Ryan Greene-

Roesel in BART’s Strategic Planning Department managed the project under the guidance of Val

Menotti and Ellen Smith. BART’s Communications, Fare Collection Engineering, Operations Planning,

Legal, Customer Service, and Treasury Departments also made key contributions.

Several forthcoming academic papers are expected from the project, including:

● Robert Hahn, Robert Metcalfe, Eddy Tam (2019). Estimating the Welfare Impacts for Public

Transit: New Experimental Evidence and Theory. Working Paper.

● Ali Arian, Chih-Wei Hsieh, Yi-Chang Chiu, Ryan Greene-Roesel (2019). Developing A Crowding

Prediction Model using A Machine Learning Approach for a Fare Card-Based Transit System.

Working Paper.

● Ali Arian, Raymond Huang, Yi-Chang Chiu, Ryan Greene-Roesel (2019). The Development and

Field Validation of a System Optimal Individualized Incentive Scheme for a Metro Transit

System. Working Paper.

Executive Summary

This report summarizes the results of BART Perks Phase II, a pilot program aimed at reducing train

crowding with incentives. One of very few international examples of projects to reduce crowding on

mass transit with incentives, Perks Phase II ran from December 2018 to June 2019, and built on the

lessons learned of a similar pilot (Perks Phase I) completed in 2017. A grant from the Federal Transit

Administration funded the project as part of a program of efforts to encourage ridership

development and more efficient capacity utilization in the BART Transbay Corridor that connects San

Francisco and Oakland via the underwater Transbay Tube.

Program Overview

Perks Phase II offered personalized incentives to a pilot group of 1,900 BART riders with the objective

of achieving a modest (approximately 5 percent) reduction in the share of travel made in congested

periods. Participants were recruited from prior Perks Phase I participants and on select station

platforms and were predominantly frequent commuters to downtown San Francisco stations. After

registering with their Clipper smart cards, participants accessed the program via BART’s website and

Official Mobile Application. Participants received limited time offers to earn points by changing their

typical departure time and cashed out their rewards by selecting from a variety of gift cards. To

support a robust program evaluation, half the participants randomly did not receive offers to shift

their commute in the first three months of the program, so their behavior could be compared to

those that did receive offers.

Incentive Offers

Participants received point offers of about $1 per trip on average for starting their journey at a

specific station (based on their most frequent entry and exit stations) and during a specific 20-minute

time window (customized for the individual rider) in the morning and/or evening commute period.

The incentivized time window was determined by algorithms, including a crowding predictive model,

that identified whether less-crowded options were available up to forty minutes before or after the

typical departure time. If a less crowded window was not identified within 40 minutes of the typical

departure time, no offer was shown, based on focus group findings that users do not like being asked

to make large shifts. Four types of offers were possible: morning-shift early, morning-shift late,

afternoon-shift early, and afternoon-shift late.

Only regular BART riders (defined as having made at least four one-way trips on BART per week on

average over the last four weeks) were eligible to receive this type of offer. Figure ES. 1 illustrates an

example shift commute offer within the BART app. Participants also received point offers designed to

encourage additional trip making on evenings and weekends, or to the airport.

Figure ES. 1: Shift Commute Offer Example

The Perks Phase II incentive approach improved upon Phase I by providing participants with

customized and dynamic, rather than static, incentive offers. Perks Phase II incentive offers were

unique to the participant and tailored to their travel history, including their station origin, average

departure time, and history of traveling during congested periods. Offers were updated monthly

based on changes in predicted congestion patterns. In comparison, Perks Phase I provided essentially

the same static incentive offering to all users regardless of whether they were already travelling at

the desired times, and without taking into account whether users would be shifted to less crowded

trains.

Results

Table ES. 1,summarizes the program results in terms of the percentage increase in the share of

participant travel made during incentivized periods, for those who received offers to shift compared

to participants who did not receive offers to shift their commute. Participants with offers increased

the share of their travel during incentivized periods by between 6 and 20 percent, depending on the

type of offer. Looking just at trips in the Transbay corridor, the range was 8-18%.

Table ES. 1: Change in Share of Participant Trips During Incentivized Periods - Average Shift

Change in Share of Travel During Incentivized Periods

Type of Offer

All Participant Trips

Trips in the Transbay Market

Morning – shift earlier

Morning – shift later

19%

14%

Afternoon/evening – shift earlier

13%

15%

Afternoon/evening – shift later

20%

18%

Note: These percentages reflect the increase in the share of travel made during incentivized periods for those receiving

the incentive compared to those who did not, from mid-December 2018 to the end of March 2019. Differences significant

at the 95% confidence level.

Figure ES.2 looks at the results another way by comparing participant travel (fraction of system

entries by time period) during the program to the 3 months prior to the program, for participants

who received each of the four types of offers. The shift late offers demonstrate a more visually

noticeable effect in causing participants to enter the system later, compared to the shift-early offers.

Figure ES. 2: Share of Participant Travel by Time of Day, Before and During the Perks Program, by

Type of Offer

Note: The “before” period refers to the three months prior to the program and the “after” period refers to the first three

months of the program. The sample sizes for panels a, b, c, and d are 661, 642, 610, and 659 respectively. Participants

may have received more than one type of offer.

The travel shift results indicate a strong response to the incentives overall and in comparison, to

Perks Phase I, which showed about a 10% reduction in travel during the dis-incentivized peak hour.

Moreover, the definition of “shift” in Perks Phase II was more robust, because it entailed shifting

from a more to a less congested time window and making time shifts of 20 to 40 minutes, as opposed

to shifting from one time period to another by any number of minutes (even one). Some of the

strength of the response can be attributed to the participation of prior Perks Phase I users. These

individuals made up about a third of Perks II program enrollment and exhibited much stronger shift

results than the new recruits. Travel behavior shifts just among new recruits ranged from 5 to 16%.

In addition to receiving offers to shift their commute time, participants received offers of between

200-500 points ($1 - $2.50) per trip for making additional BART trips on selected evenings and

weekends and to the airport. Among eleven such offers evaluated, five resulted in statistically

significant and positive increases of between 13 to more than 100% in the incentivized type of trip.

Participant Characteristics and Feedback

More than half of participants (about a thousand) responded to a survey administered in mid-April

2019 to gauge their feedback about the program, barriers to shifting commute behavior, and their

demographic characteristics. The results indicated that most participants (about 70%) were satisfied

with the program, a similar share as Perks Phase I. Overall, satisfaction was strongly related to the

amount of rewards received. Top areas of feedback included requests for expanded ways to earn

points, better notification of new offers through in-app push notifications, and different types of

rewards (besides gift cards, and especially Clipper value) and a desire to be rewarded on an ongoing

basis for riding BART, rather than just receiving limited time offers. Regarding demographic

characteristics, the survey showed that the following groups were underrepresented compared to all

BART riders: those identifying as non-white and non-Asian, low-income households, non-English

language speakers and those without a smartphone. As expected based on the recruitment

approach, Perks participants more closely reflected the demographic makeup of BART ridership to

downtown San Francisco, which tends to be more affluent than BART’s overall ridership.

Cost Effectiveness and Cost to Scale

In Perks Phase II, the incentive cost per shifted trip varied over the course of the program but was

approximately $1 overall, a significant improvement over the incentive cost of $10 per shifted trip in

Perks Phase I.

The greater degree of efficiency was achieved primarily by rewarding only behavior

change (e.g. change from baseline travel behavior) rather than rewarding pre-existing behavior as

was done in Perks Phase I, and by expanding the eligible windows for time shift.

The study team undertook a simulation of the cost to scale the Perks program to achieve an

approximately 5 percent reduction in a measure of system crowding called the total crowding score

(TCS), which represents the aggregate amount of crowding (people per train car) occurring on BART,

with higher weight given to more severely crowded conditions. Achieving this reduction would

require a program enrollment of between 30,000 and 75,000 users (assuming a range of 10 to 20%

uptake of the incentive offers), and would cost about $1.9 million per year including $1.2 million

annually in incentives, plus an additional $650,000 to cover program staffing, customer service,

The percent change was calculated by comparing the share of participant trips that were of the incentivized type among

those who received the offer compared to those who did not. For example, if 3% of trips made by those who received a

given offer were of the incentivized type, and 1% of trips made by those who did not receive the offer were of the

incentivized type, then the total percent change in trip making attributable to the incentivize would be calculated as (3% -

1%) / 1%, or a 200% increase.

Note that this figure relates only to points earned for shift commute offers, and not to “extra reward” or survey offers.

marketing and research, and information technology support. This analysis assumes that all enrolled

users would behave similarly to the 1,900 users enrolled in the Perks pilot.

A full cost-benefit analysis was outside the scope of the study, but the following simple comparisons

suggest the program is likely to be cost effective if scaled up:

● Train car comparison: A scaled up program that would reduce crowding (TCS) by 5% would

free up an equivalent of approximately 30 train cars for an annual program cost of

approximately $1.9 million. Purchasing an equivalent amount of train car capacity (about 30

cars) would cost approximately $6m annually (a new train car costs about $200,000 annually,

with a $5 million up-front purchase cost over a useful life of 25 years).

● Backfilling comparison: Assuming each shifted trip frees up space for another fare paying

passenger during peak times, the $1 per shifted trip figure compares favorably with the

approximately $4 average fare paid by the typical commuter who might backfill the space.

Summary Findings and Recommendations

BART Perks Phase II was one of very few international examples involving the use of personalized

incentives to reduce crowding on mass transit. Perks II was unique in that incentive offers were

customized to the individual and based on predicted congestion levels, as determined by a crowding

predictive model. Perks II also featured a robust case-control evaluation approach that measured

changes in the travel behavior of participants receiving offers and compared them to similar

participants who did not receive offers at the same time. Recommendations are as follows:

Strong results justify continued exploration of incentive programs to manage congestion. Perks

Phase II re-affirmed the core finding of Perks Phase I that meaningful travel behavior changes can be

accomplished through incentive programs. The program also appears to be cost effective based on a

high-level analysis, suggesting that continued investment in incentive programs is worthwhile. Some

uncertainty remains regarding whether the strong results would translate to the much larger

population (35,000 – 70,000) of BART riders necessary to scale up the program. Some Perks

participants may have self-selected into the program based on an interest in incentives and

willingness to shift. Whether enough such individuals could be found for a scaled-up program is

unknown.

The Perks platform can be leveraged to meet a range of agency goals in addition to reducing train

crowding. Once established, a program like Perks could support multiple agency goals beyond

managing train congestion. Promising future uses include:

● Encouragement of additional weekend, evening, and airport travel: Perks II showed that

incentives to encourage more evening and weekend transit trips can result in statistically

significant increases in trip making. Future programs could further explore this potential.

● Management of station crowding. Station crowding is a major concern at the downtown San

Francisco stations, and significantly increasing capacity would be very costly. Perks II did not

explore reduction of station crowding, but future programs could examine whether station

crowding can be reduced through incentives to shift to adjacent stations in addition to the

time of travel.

● Improved customer communication. The travel data provided by Perks could be leveraged to

support more tailored and customized communications. For example, in the event of a major

station closure, BART could leverage Perks data to inform Perks customers who typically travel

to the affected station at that time.

Future programs should explore even more precise targeting of congested conditions. Incentive

offers in Perks Phase II were updated monthly based on average congestion predictions for the

upcoming month. The software system had the technical capability of updating offers daily, but the

study team judged that too much variation in the offers could be confusing, especially given that offer

updates were communicated over email rather than through push notification. Future programs

could explore whether riders will respond to offers that change based on real time congestion. This

could allow even more precise targeting of congested conditions and help ensure that when riders

make travel shifts that they enjoy a congestion reduction benefit. About 73% of Perks participants

said they frequently or sometimes experienced less crowding when entering the station at the

designated time; this figure could be increased if incentives were more closely matched to real-time

congestion patterns.

Even without incentives, BART can help improve conditions for riders by providing real-time updates

in expected crowding conditions. More than 90% of Perks participants said they would like BART to

provide predictions of train crowding.

Future programs should consider incorporation of app-based incentive and crowding notifications.

A top request from Perks user surveys was for in-app push notifications to alert riders of new

incentive offers (these were communicated only by email during the program). Future programs

should consider incorporation of such notifications to maximize rider awareness of incentive offerings

and crowding conditions, especially if offers are to be updated more frequently in response to real

time variations in congestion.

Future programs should examine and consider ways to ensure fair distribution of costs and benefits

among riders of different incomes. Two-thirds of BART trips start or end on Market Street in San

Francisco, the area where BART crowding is most severe. Perks participants reflected the

demographic make-up of BART’s ridership to these stations, which tends to be more affluent

compared to BART’s overall ridership for the entire system. While workers of all incomes could

benefit from congestion reduction, future programs will need to carefully consider and address any

issues related to the potential for focusing monetary rewards on high income populations.

Future programs should seek to include Clipper value as a reward offering. Most Perks participants

were satisfied with choosing from a variety of electronic gift cards for their rewards, but many

expressed a preference for receiving Clipper smart card value instead of cash rewards or gift cards.

Future programs should seek to offer Clipper value as a reward. This will be more possible once the

next generation regional Clipper technology is available.

A dedicated source of ongoing funding is necessary to scale the program. Scaling the Perks program

to provide a system-wide crowding reduction is expected to cost about $1.9 million annually. Grant

funding or merchant partnerships could defray some of these costs, but would not remove the need

for a stable, ongoing source of funding. BART’s operating budget is very constrained, with many

potential uses for existing and new revenue sources, so new sources of funding would need to be

identified.

Introduction

The report presents the results of BART Perks Phase II, a pilot program that tested the effectiveness

of using incentives to reduce crowding on BART. It ran from mid-December 2018 to June 2019, and

was preceded by a similar pilot (Perks Phase I, Sept 2016 - March 2017). The report summarizes

program results, compares them to the results of the Phase I pilot, and provides recommendations

for future programs.

Background

Between 2004 and 2016, ridership on the San Francisco Bay Area Rapid Transit (BART) system

increased by about 40% overall and 75% in the Transbay corridor connecting San Francisco and

Oakland via the underwater Transbay Tube (Figure 1), due primarily to strong employment growth in

downtown San Francisco. Beginning in 2017, ridership began to decline in the evenings and

weekends, but heavy congestion during commute periods remains the norm.

Figure 1: BART System Map Highlighting the Transbay Corridor

BART is working to alleviate train congestion by expanding its fleet of rail cars, which will allow longer

trains with more standing room, and through implementation of the $3.5 billion Transbay Corridor

Core Capacity Project, which will increase peak hour train frequency in the Transbay tube by about

45% by 2028. The Transbay Corridor Core Capacity Program is currently in the Engineering phase of

the Federal Transit Administration's (FTA) Capital Investments Grants (CIG) Program.

Participants in the CIG program are eligible to compete for FTA Pilot Program for Transit Oriented

Development Planning Grant funds, intended to encourage economic development and ridership in

corridors slated for major transit investments. BART received a $1.1 million TOD Planning Grant

Development grant in 2016 for developing ridership in the Transbay corridor.

The grant funded two sub-projects: The first involved encouraging commercial development in

downtown Oakland through creation of the Downtown Oakland Specific Plan, to maximize corridor

throughput by leveraging existing reverse commute capacity. This report focuses on the second sub-

project funded by the grant (Perks Phase II), which aimed to reduce congestion and maximize

throughput in the peak direction along the corridor by incentivizing riders to shift their travel to less

crowded times.

BART Perks is one of only a few international examples of projects using incentives (tied to smart card

data) to reduce crowding. One prominent example is the Singapore Land Transportation Authority’s

Travel Smart Rewards program), which awarded points for riding during shoulder-peak times, and

also provided discounted off-peak rides and employer financial incentives to support schedule

flexibility.

An early evaluation found a 7.5% reduction in the average share of trips made during the

peak periods during the program compared to beforehand among participants; most of this shift

seems to have been to the few minutes just before the peak hour.

BART Perks Phase I, which was

modeled closely on the Singapore program and used the same underlying software platform,

produced a similar result of about 10% reduction in the share of peak hour travel.

Singapore Land Transportation Authority. Travel Smart. https://www.lta.gov.sg/content/ltaweb/en/public-

transport/mrt-and-lrt-trains/travel-smart.html. Accessed July 18, 2017.

Pluntke, C., and B. Prabhakar. INSINC: A Platform for Managing Peak Demand in Public Transit. JOURNEYS,

Land Transport Authority Academy of Singapore, 2013, pp. 31–39. https://web.stanford.edu/;balaji/papers/13INSINC

.pdf

San Francisco County Transportation Authority and the Bay Area Rapid Transit District, 2018. Evaluation Findings from

the BART Perks Test Program. https://www.sfcta.org/sites/default/files/2019-03/Lessons%20From%20Perks%20-

%20Eval%20Report.pdf. Accessed July 25, 2019.

BART Perks Phase II

Program Overview

BART Perks Phase II offered incentives to a pilot group of 1,900 BART riders with the objective of

achieving a modest (approximately 5 percent) reduction in the share of travel made in congested

periods. A secondary objective was to encourage travel during times when BART has low ridership or

excess capacity (evenings, weekends, and airport trains). After registering with their Clipper smart

cards, participants could access the program via BART’s website and BART Official Mobile Application.

The program provided limited time incentive offers tailored to the participant and allowed

participants to cash out their rewards by selecting from a variety of gift cards. The main program

objective was to test the effectiveness of using incentives to reduce crowding on BART.

Key Differences from Phase I

BART Perks Phase I was implemented in September 2016 with a grant from the Federal Highway

Administration’s Value Pricing Program. The firm (Urban Engines) that offered the underlying

software behind Perks Phase I decided to discontinue their product. To implement Perks Phase II,

BART worked with a new firm (Metropia, Inc.) to modify their incentives platform, which had been

developed for application to roadway travel, to suit a transit environment and to benefit from the

lessons learned of Perks Phase I.

In comparison to Perks Phase I, BART Perks Phase II:

● Enrolled 1,900 people compared to 18,000, to allow small-scale testing of the new platform,

which had never been tried in a transit context, unlike Perks Phase I which had been

successfully deployed at scale in Singapore.

● Enrolled by invitation only. The Perks Phase II recruitment was carefully controlled to ensure

sufficient participation by the target market (peak hour commuters) within the small sample

of enrollees.

● Deployed through a mobile app: Phase II was deployed through the BART.gov website and

the BART Official Mobile application. Perks Phase I used a separate, custom-built website

and required separate login credentials.

● Featured customized and dynamic incentive offerings. Perks Phase I offered essentially the

same incentives to all participants for the duration of the pilot program

, whereas Perks

Phase II incentives were customized to the individual participant, and incentive offers were

updated at least monthly based on changes in congestion levels and other factors.

● Provided limited-time offers rather than ongoing rewards. All Perks Phase II offers were

structured as limited-time offers targeted at specific individuals, compared to Perks Phase I

which provided all participants with a minimum of one point for every mile travelled on

BART. Perks Phase II participants did not receive any base rewards for general BART travel.

The full Perks Phase I Evaluation Report is available at https://www.sfcta.org/projects/bart-perks-test-program.

An exception were the “Bonus Boxes” offered during Perks Phase I, some of which were offered only to BART Riders

travelling in the crowded Transbay market. See the Perks Phase I evaluation report for details.

● Tested occasional promotional offers for weekend and evening travel, whereas Perks

Phase I was focused exclusively on reducing peak period travel.

● Paid incentives out through gift cards, whereas Perks Phase I participants were paid

exclusively by PayPal. Limitations with the current Clipper card contract prevented use of

Clipper as the payment mechanism for both pilots.

● Included a case-control approach. Half of the participants in Perks Phase II did not receive

offers to shift their commute during the first three months of the program, allowing

comparison with those who did receive offers, and providing a robust basis for analyzing the

program effectiveness (all participants received offers for evening, weekend, and airport

travel throughout the program). Because Perks Phase I did not include a case control, the

results of the program could not be assessed with the same degree of confidence.

Remainder of this Report

The remainder of this report is organized as follows:

● Chapter 1: Experimental Design and Recruitment

● Chapter 2: Software System and Incentive Approach

● Chapter 3: Response to Incentives

● Chapter 4: Participant Characteristics and Feedback

● Chapter 5: Cost Effectiveness and Cost to Scale

Chapter 1: Experimental Design and Recruitment

Experimental Design

Perks II was designed to ensure that any participant travel shifts detected could be attributable to the

incentives program and not to exogenous factors. To achieve this, program participants were

randomly assigned into two groups upon enrollment: the first group received offers to shift at the

beginning of the program, and the second group began receiving offers to shift after the first three

months. Throughout the program, both groups received additional, ongoing offers to earn extra

rewards by answering survey questions or for using BART during selected evenings or weekends, or

to the airport.

This approach is more robust than Perks Phase I, which did not feature a control. Instead, behavior

change was evaluated by comparing travel patterns before, during, and after the program, and to all

BART riders. While the evidence from Perks I strongly suggested that the behavior shifts identified

during the program were attributable to the incentives, the influence of exogenous factors could not

be ruled out.

The program experimental design was also setup to test whether distribution of a $5 sign-up bonus

affected enrollment and engagement, and whether prior participation in Perks Phase I made a

difference in the degree of travel behavior shift shown. These objectives were achieved by randomly

providing half of participants with the $5 sign-up bonus, and by recruiting Perks Phase I participants

in addition to new participants for the experiment.

Recruitment

Recruitment approach

The Perks recruitment aimed to achieve the following:

● Enroll regular BART commuters who typically travel to downtown San Francisco during

congested times.

● Enroll about 2,000 participants, to ensure differences in behavior among participants with

offers to shift and participants without could be detected with statistical significance.

● Randomly recruit about half of participants into the early and delayed offer groups.

● Randomly assign about half of participants to receive a $5 sign up incentive.

● Include some Perks Phase I participants in the recruitment, so their behavior could be

compared to those who had not participated in the first Perks.

Participants were recruited from two sources:

● Email recruitment from Perks Phase I participants: A subset of those who participated in Perks

Phase I opted in to receiving notifications from BART about future incentive programs. The

study team analyzed the travel histories of these individuals (as recorded during Perks Phase I)

and identified those who made at least 50 percent of their BART trips from downtown

Embarcadero, Montgomery, or Civic Center stations.

An email invitation to join the Perks

Phase II program was sent to a random subset of qualifying individuals.

● Distributing flyers at downtown San Francisco stations: Twelve people distributed flyers

advertising the program at the Embarcadero, Montgomery and Civic Center stations in

downtown San Francisco from 8:00 – 9:30 AM on December 13th, 2018, to coincide with peak

commute periods.

Those invited to participate in the program (either via email or flyer) were provided with a unique

sign-up code. The codes were evenly distributed into four groups:

● $5 sign-up incentive, immediate offers to shift

● No sign-up incentive, immediate offers to shift

● $5 sign-up incentive, delayed offers to shift

● No sign-up incentive, delayed offers to shift

The codes in these four groupings were distributed randomly when handing out flyers and sending

emails. Figure 2 presents the flyer.

Downtown San Francisco BART stations were the focus for recruitment because these stations have the highest

commuter ridership and trains servicing them experience the highest levels of congestion in the BART system. One of the

downtown stations (Powell Street) was not included in the recruitment because it also serves a large number of trips for

non-commuter purposes such as shopping, recreation, and tourism.

Figure 2: Recruitment Flyer (Front and Back)

Recruitment results

As occurred with Perks Phase I, enrollment targets were achieved rapidly with minimal outreach.

Approximately 1,900 individuals were recruited within a few days of the date of the flyering event,

and the study team decided to close the enrollment at that point. Ultimately, 63% of participants

were recruited through flyering and 36% through email, and 29% of those invited to participate in the

program signed up. The uptake rate was essentially the same among those who received an email

invitation (28%) versus a flyer (30%) and among those who received a $5 sign up incentive (28%)

versus no incentive (31%). In other words, neither the sign-up incentive nor the method of outreach

had a discernable effect on the likelihood of enrollment. The recruitment also resulted in a nearly

even split in enrollment within the early and delayed offer groups. Prior to the program start, these

groups had nearly identical travel behavior (Figure 3), indicating that any changes in behavior during

the program can be attributed to the incentives.

Figure 3: Travel Behavior of Early and Delayed Offer Groups Prior to Program

Note: Compares the percentage of system entries occurring in each time period in the three months prior to the

program, between those who received an early offer to shift and those who did not receive offers to shift until later in the

program.

Chapter 2: System Design and Incentive approach

System Software Design

The Perks Phase II software system consisted of front-end systems developed by BART with

consultant support (including the BART.gov website and BART official mobile applications) that

communicated with several software modules provided by a vendor (Metropia) and BART databases.

These systems are described below and diagrammed in Figure 4.

Modules provided by BART included the following:

● BART Perks web page and BART official mobile application: These are the front-end systems

that interacted with the user.

● Fare gate entry and exit database: After users opted-in to the program, data from BART’s fare

gate entry and exit database was provided to the Metropia account management module for

the purpose of calculating incentive offers and awarding points.

● Single Sign-On (SSO): This module is BART’s credentialing system, which stored participants

usernames and passwords, to authenticate users when they login to the Perks system.

● Passenger flow model: BART’s Passenger Flow Model (PFM) calculated historic train loads

based on train schedules and system entries and exits. This information was passed to

Metropia’s crowding prediction model, which dynamically updated its crowding predictions

based on historic trends.

Modules provided by Metropia included the following:

● Perks2 Application: This module consolidated all the data needed for the Perks web page and

mobile application, as well as a gift card redemption engine and administrator dashboard

interface provided by third party vendors.

● User database: This included a directory of users and their trip records obtained from BART’s

Clipper data. This information was used to create personalized incentives and communicate

with participants.

● Crowding Prediction Module: This module consisted of an algorithm that predicted train

crowding levels on a pre-defined frequency, to inform the calculation of incentives.

● Incentive calculation engine: This module integrated predicted crowding levels with riders’

behavior and trip data and calculated personalized incentive offers.

Figure 4: Structure of Perks Software Systems

The Perks II system design improved upon Perks Phase I in several ways. Perks II:

● Used BART’s credentialing system. Perks Phase I required users to generate a new username

and password to access Perks, whereas Phase II used a standard BART user name and

password that can be used for other BART purposes beyond Perks.

● Offered a package of features rather than a single-purpose website. Phase I was offered

through a stand-alone, single purpose website. Perks Phase II was offered through the BART

official mobile application, which also includes real-time train departures, trip planning and

station information, thereby providing customers with multiple reasons to download and

engage with the app.

● Provided flexibility for vendor substitution if needed. Perks Phase I was licensed from a third-

party vendor, and when the vendor stopped offering the service, an entirely new software

system was needed. Perks II was setup so that BART owns the front-end interfaces (website

and mobile app), which communicate with vendor databases via application programming

interfaces. If a change in vendor had been necessary, BART had the flexibility to maintain the

same front-end user experience with back end databases provided by a new vendor.

The most significant improvement of Perks II over Perks I was the inclusion of customized and

dynamic incentive offers based on the user’s travel history and predictions of expected crowding on

BART based on a crowding prediction model developed as part of the project. Perks Phase I provided

the same static incentive scheme to all users, and for simplification purposes, assumed that

congestion patterns do not vary spatially and do not change over time.

Incentives and User Interface

Perks participants could receive three types of offers:

● Shift commute offers: These provided users with points for entering BART up to forty minutes

before or after their typical entry time, to reduce travel during congested times.

● Extra reward offers: These offers provided users with points for making evening, weekend,

and airport BART trips at specific times and for specific stations.

● Survey offers: Users received points for answering a single survey question offered through

the mobile app. These were primarily designed to encourage engagement with the app. Users

also received points for completing a longer survey administered in April via Qualtrics.

Offers were communicated to users following their enrollment in the program by email, and users

could view their offers by logging into the Perks section of the BART website or mobile application.

Unlike a traditional rewards program, and in contrast to Perks Phase I, these offers were provided for

a limited time only, and users did not receive points for every trip taken on BART. Offer types are

discussed in more detail below.

Shift commute time offers

Shift commute offers provided users with points (ranging from 50 to 200 per trip) for entering their

most frequently-visited station at a specific 20-minute time window during the morning and/or

evening commute period. The time window shown was up to 40 minutes before or after their typical

entry time and optimized so that if the individual entered the station at the incentivized time, net

crowding on BART would be reduced. Users could receive up to four commute-related point offers

(shift early AM, shift late AM, shift early PM, shift late PM). Only regular BART riders (defined as

having made at least four one-way trips on BART per week on average over the last four weeks) were

eligible to receive this type of offer. The offer particulars (amount, timing, and station) were

determined by an algorithm that drew upon the participants travel history and the crowding

predictive model. Figure 5 illustrates an example shift commute offer.

Figure 5: Shift Commute Offer Example (Page 1 and 2)

To identify the right incentive time windows, the Perks algorithm identified the user’s typical

departure time and then calculated the predicted crowding reduction benefit of shifting this user to

one of the adjacent time periods. If no benefit would occur from shifting the user, then no offer

would be shown. Similarly, no offer was shown if achieving crowding reduction would require the

user to make larger shifts (beyond up to 40 minutes from their average departure time). This was

based on feedback from user focus groups that individuals did not wish to make large shifts in their

commute and perceived the program negatively if they thought that large shifts would be required.

The algorithm assumed that everyone who receives a point offer shifts to the incentivized window

and calculated the optimal number of people to shift to avoid over-congesting the incentivized

window.

Extra Reward Offers

All Perks participants received point offers designed to encourage additional trip making on evenings

and weekends, or to the airport. Table 1 lists the offers provided during the program. Participants

were equally and randomly divided into two groups (A and B) for the purpose of evaluating the

effectiveness of these offers. Each group A and B was comprised of half of the early offer and half of

the delayed offer group described in the prior section. In the first two and a half months of the

program, Groups A and B received the same extra reward offers. Beginning in March, 2019, different

offers were provided to each group at different times so their effectiveness could be evaluated.

Offers ranged between 200 to 500 points ($1 – 2.50) per qualifying trip. Participants could make no

more than 3 reward trips in a given day.

Table 1: Extra Reward Offers

Group A

Group B

Mid-Dec 2018

Take BART to the airport

January 2019

Take BART to the airport

February 2019

Enjoy the Bay Lights – Exit Embarcadero

Station on weeknights after 7 PM

Enjoy the Bay Lights – Exit Embarcadero

Station on weeknights after 7 PM

Ride BART with your sweetheart on

Valentine’s weekend

Ride BART with your sweetheart on

Valentine’s weekend

March 2019

Enjoy the Macy’s Flower Show - Exit

Powell Street Station March 30

/31

Cheer the St Patrick’s Day Parade – Ride

BART on March 16

Oakland Art Murmur Exit 12

or 19

Street Oakland Stations on Friday March

April 2019

Ride BART to the A’s - Exit Coliseum

Station April 20

or 21

Enjoy the Macy’s Flower Show - Exit Powell

Street Station April 6

& 7

Explore the Bay! Ride BART on Saturdays

in April

Spring getaway – ride BART to the airport

May 2019

Spring getaway – ride BART to the

airport

Explore the Bay! Ride BART on Saturdays in

May

Oakland Art Murmur Exit 12

or 19

Street Oakland Stations on Friday March

Ride BART to the A’s - Exit Coliseum Station

May 4

, 5

, 25

&26th

Group A

Group B

June 2019

Explore the Bay! Ride BART on Saturdays

in June

Spring getaway – ride BART to the airport

Point redemption

Users viewed their point history and redeemed points by logging into the Perks section of the BART

website or mobile application. Upon redemption, the user instantly received a gift-card code via

email. Users selected from among ten gift cards including pre-paid Visa, Amazon, Target, iTunes,

eBay, Starbucks, Walmart, Best Buy, Sephora, or the Tango gift card (redeemable at more than 60

additional retailers such as Google Play, Nordstrom, Pottery Barn, REI, Barnes and Noble, and CVS,

and with more than fifteen charities). Gift card options were selected based on the top-ten most

searched for gift cards nationally according to WalletHub.com, with some modifications to prioritize

gift cards that provide the most flexibility and were readily available through an off-the-shelf provider

of national gift card choices. The team did not pursue gift card options with local merchants for the

pilot period, but such options could be pursued for future deployments. Figure 6 shows the point

redemption page.

Figure 6: App Redemption Page

Chapter 3: Response to Incentives

This chapter describes the user response to incentives, including the number of points accumulated,

gift cards redeemed, and travel shifts made in response to point offers.

Point Accrual and Redemption Patterns

Perks users accumulated about $23,000 in point value over the course of the program, or

approximately $12 per participant over a six-and-a-half-month period ($1.80/participant/month on

average). Looking just at rewards associated with commute offers, this figure drops to about $7 per

participant total. Table 2 shows the distribution of points by user and Table 3 shows the share of

rewards by type.

Table 2: Share of Users by Reward Value (Cumulative Over the Program)

Dollar Value of Points Accumulated

Share of Participants

Greater than $50

$30 - $49

$10 - $29

43%

$5-$9

32%

Less than $5*

19%

*Point values of less than $5 were not redeemable, so effectively these individuals did not receive any rewards.

Table 3: Share of Rewards by Reward Type

Type of Offers

Point Value per Offer

Share of Total Rewards

Shift commute

50 – 200 points ($0.25 - $1.00)

56%

Extra offers

200 - 500 points ($1.00 - $2.50)

22%

Survey question

5 points ($0.03)

1,000 points ($5.00)

20%

Note: Figures do not total to 100% due to rounding.

As of a month following the end of the program, $13,400 of the total $23,000 of rewards had been

redeemed. This represents about 80% of the value that was eligible to redeem (after subtracting out

points less than 1,000, the minimum threshold for redemption). Among those who redeemed

rewards, Amazon was the most popular gift card choice by far. Figure 7 shows the percentage of

program rewards by type of reward.

Figure 7: Share of Redeemed Value by Type of Card

Response to Shift Commute Incentives

This section summarizes the participant response to receiving offers to shift their commute time.

Participant’s response to these offers was evaluated by comparing their travel times to the travel

times of participants who did not receive offers to shift their commute during the same month, and

by comparing participant travel behavior before and during the program. The evaluation results

reflect the first half of the program (mid December 2018 through end of March 2019). For the

remainder of the program (April to June 2019), offers to shift were provided to most participants,

making the program results from that period more difficult to ascertain.

Overall Shift Results

Participants increased the share of their travel during incentivized periods by between 6 and 20%,

depending on the type of offer (Table 4). Looking just at trips in the Transbay corridor, the range was

8-18%. These percentages reflect the increase in the share of travel made during incentivized periods

for those receiving the incentive compared to those who did not. These results are substantially

higher than the shift figures calculated for Perks Phase I, which showed about a

10% reduction in travel during the dis-incentivized peak hour.

58%

12%

11%

10%

Amazon Gift Card

Prepaid Visa Card

Starbucks Gift Card

Target Gift Card

Sephora Gift Card

iTunes Gift Card

eBay Gift Card

Walmart Gift Card

BestBuy Gift Card

Table 4: Change in Share of Participant Trips During Incentivized Periods - Average Shift

Change in Share of Travel During Incentivized Periods

Type of Offer

All Participant Trips

Trips in the Transbay Market

Morning – shift earlier

Morning – shift later

19%

14%

Afternoon/evening – shift earlier

13%

15%

Afternoon/evening – shift later

20%

18%

Note: These percentages reflect the increase in the share of travel made during incentivized periods for those receiving

the incentive compared to those who did not, from mid-December 2018 to the end of March, 2019. These results are

statistically significant at the 95% confidence level.

Figure 8 looks at the results another way, by comparing participant travel (fraction of system entries

by time period) during the program to the 3 months prior to the program, for participants who

received each of the four types of offers. The shift late offers demonstrate a more visually noticeable

effect in causing participants to enter the system later, compared to the shift-early offers.

Figure 8: Share of Participant Travel by Time of Day, Before and During the Perks Program, by Type

of Offer

Note: The “before” period refers to the three months prior to the program and the “after” period refers to the first three

months of the program. The sample sizes for panels a, b, c, and d are 661, 642, 610, and 659 respectively. Participants

may have received more than one type of offer.

The travel shift results indicate a strong response to the incentives overall and in comparison, to

Perks Phase I, which showed about a 10%reduction in travel during the dis-incentivized peak hour.

Moreover, the definition of “shift” in Perks Phase II was more robust. In Perks Phase II, “shift”

entailed moving from a more congested to a less congested time window, as determined by a

crowding predictive model and associated algorithm, and shifts were made of between 20 and

40 minutes away from the individuals prior average departure time. The algorithm also checked to

ensure that moving an additional trip to the adjacent window would not just transfer congestion

from one time period to another. In Perks Phase I by contrast, “shift” was defined as moving a trip

from the 7:30 – 8:30 time window to any other period in the morning, regardless of the difference in

departure time (e.g. traveling at 7:29 A.M. would be considered a shift), or whether the adjacent time

window and train were congested or not. This difference can be seen when comparing before and

after plots of Perks Phase I and II. Phase I (Figure 9) shows a spike in travel just before and after the

7:30 – 8:30 time window, whereas Phase II (Figure 8) shows a smoother shifting of travel in the

desired direction.

Figure 9: Inbound A.M. Transbay Participant Trips Before and During BART Perks Phase I Program

Comparison of Prior Perks Participants to New Participants

About one third of Perks Phase II participants were recruited from a list of Perks Phase I participants

who had previously opted-in to receive updates from BART about future incentives programs. Their

choice to opt-in to future communications suggests a high level of interest in incentive programs,

potentially translating into a higher propensity to make travel shifts. To investigate this, the travel

behavior of prior Perks participants was compared with new recruits. Table 5 presents the results and

shows that prior Perks participants were indeed significantly more likely to make travel behavior

shifts, especially in the morning. These findings suggest that the behavior change results among prior

Perks participants are higher than what would be expected for typical participants should the

program be scaled up.

Table 5: Comparison of Behavior Change Between Prior Perks Participants and New Recruits

Change in Share of Travel During Incentivized Periods

Type of Offer

Prior Perks Phase I

Participants

Non-Perks Phase I Participants

Morning – shift earlier

19%

Morning – shift later

31%

Afternoon/evening – shift earlier

21%

15%

Afternoon/evening – shift later

27%

16%

Note: Differences significant at the 95% confidence level.

Evaluation of Morning-Only Offers

During the month of April, a program variation was tested in which participants only received offers

to shift their commute during the morning. This was done to establish whether encouraging people

to change their AM commute time also changes their PM commute time. No clear evidence was

found to suggest that people who commuted earlier or later in the AM time as a result of the offers

change their PM commute time.

Response to Evening, Weekend, and Airport Incentives

In addition to receiving offers to shift their commute time, participants also received offers

encouraging them to make additional BART trips on selected evenings and weekends and to the

airport (referred to as “Extra Rewards” offers). These Extra Reward offers were not advertised by

email to participants, to avoid distracting from the commute shift offers, which were the main focus

of the program. Participants could discover the extra offers by logging into the Perks program

interface and scrolling down to see additional offers listed below any offers they received to shift

their commute time. Table 6 lists a subset of the offers (those that had been provided to all

participants as of mid-April 2019 when the participant survey was fielded) and shows that with the

exception of the “Take BART to the Airport” offer (which ran for almost two months, longer than

other offers), only 18-21% of participants noticed the offers they were given

Table 6: Which of the Following Offers to Earn Points Do You Remember Seeing?

Offer Name

% Reporting they saw the offer

Enjoy the Bay Lights! Exit Embarcadero after 7 pm

18%

Ride BART with your Sweetheart: Ride BART Feb 16 and 17

18%

Take BART to the airport: Exit from SFO or OAK

56%

Enjoy the Macy's Flower Show: Exit at Powell St. Station

21%

Enjoy the Bay Lights! Exit Embarcadero after 7 pm

18%

Note: N = 1017. Survey was distributed in April 2019, so offers made in May and June were not included. List

only includes offers shown to both A and B groups.

As described in the prior chapter, the same Extra Rewards offers were initially provided to all

participants. The effectiveness of these early offers is not evaluated in this report, because of lack of

comparison group. Beginning March 2019, participants were randomly assigned to an A and B group

(different from the groups used to evaluate the commute offers), and each group received different

offers at different times for the remaining months of the program, so their effectiveness could be

evaluated. Table 7 lists the offers and their value.

Table 7: Extra Reward Offers (March 2019 – June 2019)

Offer Title

Point Value

Offer Rules

Oakland Art Murmur

(Encourage evening travel)

300

Exit 12

or 19

Street Oakland Stations on Friday

March 1st, 5-9 PM

500

Exit 12

or 19

Street Oakland Stations on Friday

May 3

, 5-9 PM

Cheer the St Patrick’s Day

Parade (encourage weekend

travel)

200

Ride BART on March 16

Enjoy the Macy’s Flower Show

(encourage weekend travel)

300

Exit Powell Street Station March 30

or 31

300

Exit Powell Street Station April 6

or 7

Explore the Bay! (encourage

weekend travel)

200

Ride BART on Saturdays in April

200

Ride BART on Saturdays in May

Ride BART to the A’s

(encourage weekend travel)

300

Exit Coliseum Station April 20

or 21

300

Exit Coliseum Station May 4

, 5

, 25

&26th

Spring getaway (encourage

airport travel)

500

Exit SFO or OAK in April

500

Exit SFO or Oak in May

Five of the offers listed above resulted in a positive and statistically significant effect on travel

behavior. Table 8 shows those five offers and their result. Effects ranged from a 13 to 169%

difference between the share of trips made in the incentivized period between the test and

comparison groups. These results indicate that incentives hold promise for encouraging additional

evening and weekend travel.

Note however, that the absolute change in trip-making associated with most of these offers was

quite small. For the “Ride BART to the A’s” incentive example, less than half a percent of the

treatment group, or 4 people, made the requisite trip (exit Coliseum station on April 20

or 21

). This

is in part because most offers were narrowly defined (exit a specific station on a specific day), and

because, as described above, most participants did not notice the offers they were given. To fully

understand the potential of programs such as Perks to encourage evening and weekend travel,

further testing would be necessary, as would better promotion of the offers to ensure they are

noticed.

Table 8: Participant Response to Extra Reward Offers

Offer Title

Offer Rules

Share of Trips

– No offer

Share of trips-

with offer

Increase attributable

to offer

Oakland Art Murmur

(Encourage evening

travel)

Exit 12

or 19

Street Oakland

Stations on Friday

May 3

, 5-9 PM

3.5%

5.4%

55%

Cheer the St Patrick’s

Day Parade

(encourage weekend

travel)

Ride BART on

March 16

6.9%

7.8%

13%

Enjoy the Macy’s

Flower Show

(encourage weekend

travel)

Exit Powell Street

Station March 30

or 31

0.9%

1.6%

67%

Enjoy the Macy’s

Flower Show

(encourage weekend

travel)

Exit Powell Street

Station April 6

1.0%

1.2%

23%

Ride BART to the A’s

(encourage weekend

travel)

Exit Coliseum

Station April 20

0.2%

0.4%

169%

Note: All differences are significant at the 95% confidence level.

Response to $5 Sign-Up Incentive

As noted previously, half of participants were recruited with a $5 sign up bonus to test whether it

would encourage higher rates of enrollment and/or higher propensity to shift. The latter question

stemmed from a Perks Phase I finding that those who received higher rewards early in the program

showed greater propensity to shift throughout the pilot.

The $5 bonus did not affect rates of program enrollment. In fact, the share of those who chose to

enroll was slightly higher among those who received no bonus. The bonus did appear to impact travel

behavior during the program, but not in the manner expected based on Perks Phase I results. Those

who received the early offers to shift their commute, and who received the $5 bonus earned 88

points less on average than those who didn’t, indicating that the bonus may have slightly decreased

their likelihood of following the offers they were given. Conversely, those who received the delayed

offers to shift and received the $5 bonus earned 129 points more on average than those who did not

get the bonus. These results suggest that the $5 bonus was helpful only in that it stimulated some

program engagement among those who did not get other forms of rewards early in the program.

Chapter 4: Participant Characteristics and

Feedback

This chapter describes participant characteristics, based on analysis of their travel trends and the

results of a participant survey distributed in April 2019 (mid-way through the program), including

participant demographic characteristics, feedback about the program, and ability to shift their

schedules. More than half of participants (about a thousand) responded to the survey.

Travel Characteristics

Both the Perks Phase I and II recruitments focused on enrolling commuters in the Transbay Corridor

by targeting outreach efforts in the downtown San Francisco Station, but the recruitment strategies

had several key differences. The Perks Phase I recruitment took place over several days, anyone could

news outlets. The Perks Phase II recruitment was done via distributing flyers for a single hour on a

single day as well as through email, was not advertised, and an individualized coupon code was

required to participate. In spite of these differences, the recruitments enrolled similar types of

individuals.

Table 9 below compares the percentage of participants who were regular commuters and travelled

regularly in the Transbay market and shows that these characteristics were very similar across the

two pilots, with a slightly higher concentration of Transbay riders in Perks Phase II.

Table 9: Share of Transbay Trips and Commuters by Perks I and Perks II

Perks I

Perks II

Share of Transbay trips

69.4%

73.8%

Share of commuters

65.7%

65%

Note: Share of Transbay trips are share among all trips. Share of commuters is number of users who had at least 8 trips

per week for a median week. Perks Phase II characteristics reflect patterns up through March 2019.

Demographic Characteristics

Perks Phase I Participants were asked to report their demographic and personal characteristics

including their ethnicity, household income, whether they speak a language other than English at

home, and whether they have access to a smartphone. Figure 10 compares their responses to those

of Perks Phase I participants, all BART riders, and BART weekday riders to downtown San Francisco

stations. It illustrates that Perks Phase I and II participants had similar characteristics, and in both

pilots, non-whites and non-Asians, low-income households, non-English language speakers and those

without a smartphone were under-represented compared to all BART riders, and to a lesser extent,

compared to BART downtown commuters. As expected based on the fact that recruitment took

place in downtown San Francisco during peak times, Perks participant characteristics match more

closely with BART commuters to San Francisco than they do with BART riders as a whole.

Figure 10: Comparison of Demographic Characteristics of Perks Phase I & II participants and all

BART riders

Notes

Perks Phase II Survey (n=1,007) administered April 2019.

Perks Phase I Survey (n=10,351) reflects the combined results of two similar surveys administered fall 2017 and spring

2018.

BART Riders – Downtown Commuters (n=1693) reflects the weighted results of the BART Customer Satisfaction Survey,

administered fall 2018, but limited to respondents who reported entering or exiting BART in downtown San Francisco

(Embarcadero, Montgomery, Powell, Civic Center Stations) and if the survey was collected on a “weekday peak” train

(train run dispatched from end of line between 5:30-8:30 AM or 3:30-6:30 PM on a weekday).

All BART Riders” reflects the weighted results of the BART Customer Satisfaction Survey (n=5,294), administered fall 2018.

All differences between the Perks Phase I, Phase II, and all BART Riders groups are statistically significant at the 95%

confidence level, with the exception of the difference between the share of smartphone users between Perks Phase I and

II. The significance of differences with the BART Riders – Downtown Commuters group was not evaluated.

61%

31%

57%

13%

27%

64%

14%

43%

65%

26%

41%

% Non White % HH income < $50k % Non English at Home % No Smartphone

Perks II Perks I BART Riders - Downtown SF All BART Riders

Customer Feedback

Satisfaction

Customer response to the Perks program was gauged via an online survey administered in early April,

2019. Responses are indicated only for the group that received early offers to shift, since the delayed

offers group would have just begun receiving such offers. The early offer group reported 65%

satisfaction (with 37% somewhat and 27% very satisfied) with the program. As shown in Figure 11

satisfaction was closely correlated with the amount of rewards received.

Figure 11: Average Cumulative Points Earned Compared to Satisfaction with the Perks II Program

Notes: Reflects cumulative points earned up to mid-April, 2019, when the satisfaction survey was administered. N=1071.

These satisfaction rates are comparable to, or somewhat lower than, results from Perks Phase I.

During Perks Phase I, two similar surveys were administered to the same group of participants. The

first (from December 2016), indicated satisfaction levels of 68%, but this rose to 78% by the time the

second-round survey was completed in February 2017.

Nevertheless, the plurality of Perks II early-offer participants who had previously participated in Perks

Phase I indicated a preference for the new program: 43% indicated they liked the new Perks better

than the old, and 22% said they liked the old Perks better. The remainder said they liked them both

the same or didn’t know.

About 600 respondents provided open ended responses about the program. Based on a randomly-

selected sample of 100 of these responses, the top comments related to:

● Desire for more/expanded ways to earn points (19% of comments)

● Desire for in-app or push notifications to ensure they were made aware of the point offers

(18% of comments)

● Desire for different types of reward (besides gift cards) (12% of comments)

500

1000

1500

2000

2500

3000

3500

Very

Dissatified

Somewhat

Dissatisfied

Neutral Somewhat

Satisfied

Very Satisfied

Average Cumulative Points

● Desire to be rewarded on an ongoing basis for riding BART, rather than having just limited

time offers (10% of comments)

In comparison, the top responses from the Perks Phase I program (based on a qualitative

classification of more than 6,000 open-ended survey comments submitted) included a desire for

expanded ways to earn points (about 30%), different types of rewards (about 20%), and a concern

that the rewards were too low (about 18%). Far fewer respondents in Phase II expressed concern

about rewards being too low (about 3%). In Perks Phase I, a typical commute reward trip earned less

than $0.10

, whereas a typical commute reward trip in Perks II earned about $1.

Customers were also asked about their satisfaction with receiving various electronic gift cards as the

form of reward. About 74% of the A group indicated “very or somewhat” satisfied. This appears to be

comparable to the satisfaction experienced with use of PayPal for Perks Phase I, although

comparisons are difficult because the Perks Phase I question was asked on a 4-point scale (Excellent,

Good, Only Fair, Poor), whereas Perks Phase I was asked on a 5-point scale. About 70% of Perks Phase

I participants rated use of PayPal for rewards “excellent” or “good.”

Responses to Commute Offers

Participants were asked about various factors that might have influenced whether they followed the

commute offer they were given.

This value is based the exchange value of points to dollars. The actual payout varied by individual, because most rewards

were paid out via a random rewards generator (Spin to Win Game), in which participants could receive nothing for their

points or a value of up to as much as $100 at a time.

Table 10 shows the results. The top barrier listed was shifting commute time (34% indicating they

could rarely or never do so). This was followed by the burden of having to check the offers and

remember when to enter the station. The offer point value not being high enough was also cited as

an issue, but not by as much of a margin. Less than a third of individuals said that they did not

experience a crowding reduction as a result of the offer. This indicates that in the majority of

instances, the shift offers functioned as intended to encourage travel at less crowded times.

Table 10: Percentage of Participants by Barriers to Following Shift Commute Offers

% Answering Rarely or Never

I can shift my typical commute time by 20-40 minutes

34%

I check my offers to see how to earn points

31%

I can remember when I need to enter the station

31%

When I enter at the designated time, I experience less crowding

27%

The offer points are high enough to interest me

26%

The offers are for a station I use frequently

Ability to Shift

Participants were also asked to list barriers to shifting. They could select multiple reasons. The top

reason for not being able to shift earlier was personal preference (presumably, habit and not wanting

to wake up early). The top barrier to shifting later was also personal preference but was followed very

closely by concerns that their employer would not allow it. These results are generally consistent with

top barriers to shifting in Perks Phase I, which highlighted personal preference as the top barrier to

shifting early and employer concerns as the top barrier to shifting late.

Table 11: Barriers to Shifting Commute Time Earlier and Later

Shift 20-40 Minutes Earlier

Shift 20-40 Minutes Later

Personal preference

55%

41%

Employer would not allow it

29%

40%

Nature of the work would not allow it

28%

33%

Child care constraints

16%

12%

Other after-work commitments

17%

21%

Parking availability at BART

17%

24%

Other (specify)

10%

Feedback Obtained Through Customer Service

The highest concentration of customer service inquiries for Perks Phase II was during the program

launch and related to a technical issue associated with program login that was resolved quickly.

Following launch, inquiries came primarily from individuals who did not receive points as expected. In

many cases, this was due to their trip being made very near the entry time cutoff for receiving points.

Several individuals also expressed confusion about the fact that the program did not reward all their

BART trips on some level, as is expected for a typical rewards program (such as for airline rewards).

This contrasts to Perks Phase I, in which most inquiries stemmed from confusion about how to

redeem points. Participants were paid via PayPal and needed to use the same email address for

PayPal as they did when registering for the Perks program. Many inquiries were generated by

customers who did not realize that their rewards were coming via PayPal, or if they used a different

email address for PayPal so could not access their rewards. In Perks Phase II, these issues were

avoided by providing participants with the ability to redeem their rewards for gift cards directly

within the application.

Chapter 5: Cost Effectiveness and Cost to Scale

This section examines the program cost effectiveness and the cost to scale up the program to achieve

a desired level of crowding reduction.

Cost Effectiveness

One metric of the program cost effectiveness is the incentive cost per shifted trip. This is estimated

by dividing the approximate number of trips shifted by the program by the cost of the associated

incentives. For Perks Phase I, this cost amounted to $10 per shifted weekday Transbay Trip and $7 for

all shifted weekday trips. These shifted trips may or may not have reduced the number of people on

crowded trains – the Perks program design assumed that any trip shifted out of the 7:30 – 8:30

period, regardless of the trip geography or the amount of shift in minutes, was beneficial.

In Perks Phase II, the incentive cost per shifted trip varied over the course of the program but was

approximately $1 overall, a significant improvement over the $10 of incentive per shifted trip in Perks

Phase I.

The greater degree of efficiency was achieved primarily by rewarding only behavior change

(e.g. change from baseline travel behavior) rather than rewarding pre-existing behavior as was done

in Perks Phase I, and by expanding the eligible windows for time shift. Moreover, trips shifted in Perks

Phase II can be more readily assumed to produce a reduction in the number of people on crowded

trains, since offers were only provided if they would produce a crowding benefit based on the

crowding predictive model and incentive algorithm behind the program.

Cost of Scaled-Up Program

Scaling up the Perks program to produce a meaningful reduction in system crowding would require

dedicated staffing and an increased incentive budget. To estimate these costs, the study team

undertook a simulation of the BART system (the technical methodology of the simulation is

documented in a separate paper available upon request). The simulation assumed that BART wishes

to achieve a 5% reduction in a measure of system crowding, called the total crowding score (TCS),

defined as the aggregated cube of density (people per train) over system segments and time periods

on a given day. This formulation gives exponentially more weight to very crowded conditions.

The analysis indicated that achieving a reduction of 5% would require a program enrollment of

between 30,000 and 75,000 users (assuming a range of 10 to 20 % uptake of the incentive offers) and

would cost about $1.9 million per year including $1.2 million annually in incentives. Additional annual

program costs (for an ongoing program) are estimated at approximately $650,000 and include

staffing costs for program oversight and management and customer service, marketing and research,

and information technology support for maintenance of front and back end software systems. This

estimate assumes continuation of existing Perks program and excludes the start-up costs associated

with developing a new program.

A full cost-benefit analysis was outside the scope of the study, but the following high-level

comparisons suggest the program is likely to be cost effective:

Note that this figure relates only to points earned for shift commute offers, and not to “extra reward” or survey offers.

● Train car comparison: A scaled up program that would reduce crowding (TCS) by 5% would

free up an equivalent of approximately 30 train cars for an annual program cost of

approximately $1.9 million. Purchasing an equivalent amount of train car capacity (about 30

cars) would cost approximately $6m annually (a new train car costs about $200,000 annually,

with a $5 million up-front purchase cost over a useful life of 25 years).

● Backfilling comparison: Assuming each shifted trip frees up space for another fare paying

passenger, the $1 per shifted trip figure compares favorably with the approximately $4

average fare paid by the typical commuter.

Potential Sources of Funding

New funding sources would need to be identified to support Perks II if scaled up as an ongoing

program. This section discusses several options for funding sources. Although core program costs

could be offset from a variety of sources, a stable, ongoing source of funds would be necessary to

ensure that predictable incentives could be provided to participants and to sustain the staffing

necessary to oversee the program.

BART Operating Revenue

BART’s operating budget is very constrained, with many potential uses for existing and any new

revenue sources. Investment in incentives would need to be weighed against competing budget

priorities. New sources of operating funding would likely need to be identified to support the

program.

Grant Funding Sources

Federal funds were used to develop the Perks Phase I and II pilots. Future state or federal grants

could be used to support the addition of new program features, such as gamification or adding

rewards based on mode of access to the station, as opposed to ongoing program operating costs.

Table 12 lists potentially relevant local, regional, state and federal grant opportunities. Many grants

explicitly disallow the use of funds for incentives, and most do not fund ongoing operating costs.

Table 12: Potential Grant Funding Sources for BART Perks

Fund Source

Agency

Eligible Uses

Available Funding

Potential

Limitations

Measure BB:

Alameda County

Transportation

Sales Tax

ACTC

The Technology

Expenditure Category

funds technology,

innovation, and

development programs

Approximately $77M

available in the overall

category, to be

distributed between

2015-2045

Must be focused on

Alameda County

residents

Measure J: Contra

Costa County

Transportation

Sales Tax

CCTA

Expenditure Category 17

includes funding for

commute alternatives

Approximately $20M

available in the overall

category, to be

distributed between

2004-2034

Must be focused on

Contra Costa

County residents

Proposition K: San

Francisco County

Transportation

Sales Tax

SFCTA

The Expenditure

Subcategory for

Transportation Demand

Management / Parking

Management funds

projects that can lead to

reduction of single-

occupant vehicle

dependence and

encourage alternative

modes of travel

Approximately $13M

available in the overall

subcategory, to be

distributed between

2003-2033

Must be focused on

San Francisco

County residents;

BART is not a

sponsoring agency

(BART would have

to partner with a

sponsoring agency

to apply for funds)

Transportation

Fund for Clean Air

BAAQMD

Funds eligible projects

that reduce on-road

motor vehicle emissions

Annual call for projects,

$10k - $1.5M per

project

Must demonstrate

impact on reducing

vehicle miles of

travel

Sustainable

Communities

Transportation

Planning Grant

Caltrans

Funds local and regional

multimodal

transportation and land

use planning projects

that further the region’s

RTP SCS or contribute to

the State’s GHG

reduction targets

Annual call for projects,

approximately $17M

per year, $50k - $1M

per project

Intended for

planning projects;

incentives for public

participation may

be ineligible

Advanced

Transportation and

Congestion

Management

Technologies

Deployment

FHWA

Funds deployment of

advanced transportation

and congestion

management

technologies

Annual call for projects,

$60M per year, up to

$12M per recipient

Must demonstrate a

congestion

reduction benefit

Mobility on

Demand Sandbox

Program

FTA

Funds activities leading

to the demonstration of

innovative mobility on

demand and transit

integration concepts

First cycle allocated

$8M in 2016, anticipate

second cycle in 2019

with a similar level of

funding

Primarily intended

to expand mobility

options

Other Funding Options

● Merchant partnerships: BART staff could solicit merchants to provide occasional contributions

to the program. The cost of additional staffing required to seek and manage partnerships

would need to be compared to the potential savings obtained through merchant donations. A

stable, ongoing source of funding for incentives would still be needed even with merchant

contributions.

● Employer partnerships / Transportation Management Agencies: BART could seek employer

partnerships and request that employers cover some of the cost of incentives for their

employees. Such contributions could be voluntary or could be structured as a method for

employers to meet requirements imposed on them through development impact agreements.

For example, many of the buildings in downtown San Francisco are part of Transportation

Management Agencies that require a financial contribution to help offset the congestion and

vehicle trip related impacts associated with commuting.