programming language research articles

Proceedings of the ACM on Programming Languages

Volume 8, Issue ICFP

August 2024

Proceedings of the ACM on Programming Languages (PACMPL) is a Gold Open Access journal publishing research on all aspects of programming languages, from design to implementation and from mathematical formalisms to empirical studies. Each issue of the journal is devoted to a particular subject area within programming languages and will be announced through publicized Calls for Papers. All accepted papers receive two rounds of reviewing and authors can expect initial decisions regarding submissions in under 3 months. The journal operates in close collaboration with the Special Interest Group on Programming Languages (SIGPLAN) and is committed to making high-quality peer-reviewed scientific research in programming languages free of restrictions on both access and use.

Subject Areas

Announcements.

ACM Updates Its Peer Review Policy

ACM is pleased to announce that its Publications Board has approved an updated Peer Review Policy . If you have any questions regarding the update, the associated FAQ addresses topics such as confidentiality, the use of large language models in the peer review process, conflicts of interest, and several other relevant concerns. If there are any issues that are not addressed in the FAQ, please contact ACM’s Director of Publications, Scott Delman .

New ACM Policy on Authorship ACM has a new Policy on Authorship , covering a range of key topics, including the use of generative AI tools.  Please familiarize yourself with the new policy and the associated list of Frequently Asked Questions .

Most Frequent Affiliations

Most cited authors, latest issue.

• Volume 8, Issue ICFP August 2024 EISSN: 2475-1421 View Table of Contents

Snapshottable Stores

Inria, Paris, France

OCamlPro, Paris, France

Université Paris Cité, Inria, CNRS, Paris, France

Beyond Trees: Calculating Graph-Based Compilers (Functional Pearl)

IT University of Copenhagen, Copenhagen, Denmark

University of Nottingham, Nottingham, United Kingdom

Recent Award Winners

Most popular, other acm journals.

Volume 2, Issue 3

September 2024

Volume 3, Issue 2

April-June 2024

Volume 56, Issue 12

December 2024

Volume 5, Issue 3

Volume 3, Issue 3

Volume 36, Issue 3

Export Citations

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Help | Advanced Search

Computer Science > Programming Languages

Title: rust: the programming language for safety and performance.

Abstract: Rust is a young programming language gaining increased attention from software developers since it was introduced to the world by Mozilla in 2010. In this study, we attempt to answer several research questions. Does Rust deserve such increased attention? What is there in Rust that is attracting programmers to this new language? Safety and performance were among the very first promises of Rust, as was claimed by its early developers. Is Rust a safe language with high performance? Have these claims been achieved? To answer these questions, we surveyed and analyzed recent research on Rust and research that benchmarks Rust with other available prominent programming languages. The results show that Rust deserves the increased interest by programmers, and recent experimental results in benchmarking research show Rust's overall superiority over other well-established languages in terms of performance, safety, and security. Even though this study was not comprehensive (and more work must be done in this area), it informs the programming and research communities on the promising features of Rust as the language of choice for the future.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Values of Inclusion
• 2020 Antiracism Task Force
• 2022 DEI Report
• Research News
• Department Life
• Listed by Recipient
• Listed by Category
• Oral History of Cornell CS
• CS 40th Anniversary Booklet
• ABC Book for Computer Science at Cornell by David Gries
• Books by Author
• Books Chronologically
• The 60's
• The 70's
• The 80's
• The 90's
• The 00's
• The 2010's
• Faculty Positions: Ithaca
• Faculty Positions: New York City
• Lecturer Position: Ithaca
• Post-doc Position: Ithaca
• Staff/Technical Positions
• Ugrad Course Staff
• Ithaca Info
• Internal info
• Graduation Information
• Cornell Learning Machines Seminar
• Student Colloquium
• Fall 2024 Colloquium
• Conway-Walker Lecture Series
• Salton 2024 Lecture Series
• Fall 2024 Artificial Intelligence Seminar
• Fall 2024 Robotics Seminar
• Fall 2024 Theory Seminar
• Big Red Hacks
• Cornell University - High School Programming Contests 2024
• Game Design Initiative
• CSMore: The Rising Sophomore Summer Program in Computer Science
• Explore CS Research
• ACSU Research Night
• Cornell Junior Theorists' Workshop 2023
• Researchers
• Ph.D. Students
• M.Eng. Students
• M.S. Students
• Ph.D. Alumni
• M.S. Alumni
• List of Courses
• Course and Room Roster
• CS Advanced Standing Exam
• Architecture
• Artificial Intelligence
• Computational Biology
• Database Systems
• Human Interaction
• Machine Learning
• Natural Language Processing

Programming Languages

• Scientific Computing
• Software Engineering
• Systems and Networking
• Theory of Computing
• Contact Academic Advisor
• Your First CS Course
• Technical Electives
• CS with Other Majors/Areas
• Transfer Credits
• CS Honors Program
• CPT for International CS Undergrads
• Graduation Requirements
• Useful Forms
• Becoming a CS Major
• Requirements
• Game Design Minor
• Co-op Program
• Cornell Bowers CIS Undergraduate Research Experience (BURE)
• Independent Research (CS 4999)
• Student Groups
• UGrad Events
• Undergraduate Learning Center
• UGrad Course Staff Info
• The Review Process
• Early M.Eng Credit Approval
• Financial Aid
• Prerequisites
• The Application Process
• The Project
• Pre-approved Electives
• Degree Requirements
• The Course Enrollment Process
• Advising Tips
• Entrepreneurship
• Cornell Tech Programs
• Professional Development
• Contact MEng Office
• Career Success
• Applicant FAQ
• Computer Science Graduate Office Hours
• Exam Scheduling Guidelines
• Graduate TA Handbook
• MS Degree Checklist
• MS Student Financial Support
• Special Committee Selection
• Diversity and Inclusion
• Contact MS Office
• Ph.D. Applicant FAQ
• Graduate Housing
• Non-Degree Application Guidelines
• Ph. D. Visit Day
• Advising Guide for Research Students
• Business Card Policy
• Cornell Tech
• Curricular Practical Training
• A & B Exam Scheduling Guidelines
• Fellowship Opportunities
• Field of Computer Science Ph.D. Student Handbook
• Field A Exam Summary Form
• Graduate School Forms
• Instructor / TA Application
• Ph.D. Requirements
• Ph.D. Student Financial Support
• Travel Funding Opportunities
• Travel Reimbursement Guide
• The Outside Minor Requirement
• CS Graduate Minor
• Outreach Opportunities
• Parental Accommodation Policy
• Special Masters
• Student Spotlights
• Contact PhD Office

Search form

You are here

Phd students.

• Luke Bernick
• Oliver Daids
• Dietrich Geisler
• Mark Moeller
• Rachit Nigam
• Oliver Richardson
• Goktug Saatcioglu

The programming languages research group at Cornell includes eight faculty and over two dozen Ph.D. students. We are proud of both our breadth and depth in this core discipline. Cornell has been known from the beginning for its research in programming languages. We have made foundational contributions to type theory, automated theorem proving, and language semantics. A more recent theme has been language-based solutions to important problems such as computer security, networking, and distributed programming. Cornell researchers have also contributed to language implementation, program analysis and optimization, domain-specific languages, and software engineering.

See the PL group's site for news and a full list of people involved in PL research.

Robert Constable  researches programming languages and formal methods in the context of type theory. The Nuprl proof assistant, developed by Constable and his group, is a dependently-typed language that can be used to describe distributed computing, as a formal specification language for computing tasks, and as a theory for formalizing topics in constructive and intuitionistic mathematics (of which classical mathematics can usually be seen as a special case). Constable is also interested in synthesizing programs and concurrent processes from proofs, developing systems that can be shown to be secure by construction, and exploring the deep connections between programming and logic.

Suggestions or feedback?

MIT News | Massachusetts Institute of Technology

• Machine learning
• Sustainability
• Black holes
• Classes and programs

Departments

• Aeronautics and Astronautics
• Brain and Cognitive Sciences
• Architecture
• Political Science
• Mechanical Engineering

Centers, Labs, & Programs

• Abdul Latif Jameel Poverty Action Lab (J-PAL)
• Picower Institute for Learning and Memory
• Lincoln Laboratory
• School of Architecture + Planning
• School of Engineering
• School of Humanities, Arts, and Social Sciences
• Sloan School of Management
• School of Science
• MIT Schwarzman College of Computing

To the brain, reading computer code is not the same as reading language

Press contact :, media download.

*Terms of Use:

Images for download on the MIT News office website are made available to non-commercial entities, press and the general public under a Creative Commons Attribution Non-Commercial No Derivatives license . You may not alter the images provided, other than to crop them to size. A credit line must be used when reproducing images; if one is not provided below, credit the images to "MIT."

Previous image Next image

In some ways, learning to program a computer is similar to learning a new language. It requires learning new symbols and terms, which must be organized correctly to instruct the computer what to do. The computer code must also be clear enough that other programmers can read and understand it.

In spite of those similarities, MIT neuroscientists have found that reading computer code does not activate the regions of the brain that are involved in language processing. Instead, it activates a distributed network called the multiple demand network, which is also recruited for complex cognitive tasks such as solving math problems or crossword puzzles.

However, although reading computer code activates the multiple demand network, it appears to rely more on different parts of the network than math or logic problems do, suggesting that coding does not precisely replicate the cognitive demands of mathematics either.

“Understanding computer code seems to be its own thing. It’s not the same as language, and it’s not the same as math and logic,” says Anna Ivanova, an MIT graduate student and the lead author of the study.

Evelina Fedorenko, the Frederick A. and Carole J. Middleton Career Development Associate Professor of Neuroscience and a member of the McGovern Institute for Brain Research, is the senior author of the paper, which appears today in eLife . Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and Tufts University were also involved in the study.

Language and cognition

A major focus of Fedorenko’s research is the relationship between language and other cognitive functions. In particular, she has been studying the question of whether other functions rely on the brain’s language network, which includes Broca’s area and other regions in the left hemisphere of the brain. In previous work, her lab has shown that music and math do not appear to activate this language network.

“Here, we were interested in exploring the relationship between language and computer programming, partially because computer programming is such a new invention that we know that there couldn’t be any hardwired mechanisms that make us good programmers,” Ivanova says.

There are two schools of thought regarding how the brain learns to code, she says. One holds that in order to be good at programming, you must be good at math. The other suggests that because of the parallels between coding and language, language skills might be more relevant. To shed light on this issue, the researchers set out to study whether brain activity patterns while reading computer code would overlap with language-related brain activity.

The two programming languages that the researchers focused on in this study are known for their readability — Python and ScratchJr, a visual programming language designed for children age 5 and older. The subjects in the study were all young adults proficient in the language they were being tested on. While the programmers lay in a functional magnetic resonance (fMRI) scanner, the researchers showed them snippets of code and asked them to predict what action the code would produce.

The researchers saw little to no response to code in the language regions of the brain. Instead, they found that the coding task mainly activated the so-called multiple demand network. This network, whose activity is spread throughout the frontal and parietal lobes of the brain, is typically recruited for tasks that require holding many pieces of information in mind at once, and is responsible for our ability to perform a wide variety of mental tasks.

“It does pretty much anything that’s cognitively challenging, that makes you think hard,” Ivanova says.

Previous studies have shown that math and logic problems seem to rely mainly on the multiple demand regions in the left hemisphere, while tasks that involve spatial navigation activate the right hemisphere more than the left. Working with Marina Bers, a professor of child study and human development at Tufts University, the MIT team found that reading computer code appears to activate both the left and right sides of the multiple demand network, and ScratchJr activated the right side slightly more than the left.

Effects of experience

The researchers say that while they didn’t identify any regions that appear to be exclusively devoted to programming, such specialized brain activity might develop in people who have much more coding experience.

“It’s possible that if you take people who are professional programmers, who have spent 30 or 40 years coding in a particular language, you may start seeing some specialization, or some crystallization of parts of the multiple demand system,” Fedorenko says. “In people who are familiar with coding and can efficiently do these tasks, but have had relatively limited experience, it just doesn’t seem like you see any specialization yet.”

In a companion paper appearing in the same issue of eLife , a team of researchers from Johns Hopkins University also reported that solving code problems activates the multiple demand network rather than the language regions.

The findings suggest there isn’t a definitive answer to whether coding should be taught as a math-based skill or a language-based skill. In part, that’s because learning to program may draw on both language and multiple demand systems, even if — once learned — programming doesn’t rely on the language regions, the researchers say.

“There have been claims from both camps — it has to be together with math, it has to be together with language,” Ivanova says. “But it looks like computer science educators will have to develop their own approaches for teaching code most effectively.”

The research was funded by the National Science Foundation, the Department of the Brain and Cognitive Sciences at MIT, and the McGovern Institute for Brain Research.

Share this news article on:

Related links.

• Evelina Fedorenko
• Anna Ivanova
• Una-May O’Reilly
• Shashank Srikant
• McGovern Institute for Brain Research
• Department of Brain and Cognitive Sciences

Related Topics

• Brain and cognitive sciences
• McGovern Institute
• National Science Foundation (NSF)
• Linguistics
• Neuroscience
• STEM education
• Computer science and technology
• Technology and society
• Computer Science and Artificial Intelligence Laboratory (CSAIL)

Related Articles

Brain’s language center has multiple roles

Localizing language in the brain

MIT-led team finds language without numbers

Previous item Next item

More MIT News

Study: AI could lead to inconsistent outcomes in home surveillance

Read full story →

Improving biology education here, there, and everywhere

Liftoff: The Climate Project at MIT takes flight

Bridging the heavens and Earth

MIT OpenCourseWare sparks the joy of deep understanding

A wobble from Mars could be sign of dark matter, MIT study finds

• More news on MIT News homepage →

Massachusetts Institute of Technology 77 Massachusetts Avenue, Cambridge, MA, USA

• Map (opens in new window)
• Events (opens in new window)
• People (opens in new window)
• Careers (opens in new window)
• Accessibility
• Social Media Hub
• MIT on Facebook
• MIT on YouTube
• MIT on Instagram

Advertisement

Visual and textual programming languages: a systematic review of the literature

• Published: 15 March 2018
• Volume 5 , pages 149–174, ( 2018 )

Cite this article

• Mark Noone   ORCID: orcid.org/0000-0002-4618-5982 1 &
• Aidan Mooney 1  

2869 Accesses

62 Citations

5 Altmetric

Explore all metrics

It is well documented and has been the topic of much research as well that Computer Science courses tend to have higher than average drop-out rates at third level, particularly so, for students advancing from first year to second year. This is a problem that needs to be addressed not only with urgency but also with caution. The required number of Computer Science graduates is growing every year, but the number of graduates is not meeting this demand, and one way that this problem can be alleviated is to encourage students, at an early age, towards studying Computer Science courses. This paper presents a systematic literature review that examines the role of visual and textual programming languages when learning to program, particularly as a First Programming Language. The approach is systematic in that a structured search of electronic resources has been conducted, and the results are presented and quantitatively analysed. This study will provide insight into whether or not the current approaches to teaching young learners programming are viable, and examines what we can do to increase the interest and retention of these students as they progress through their education.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Applying Hybrid Programming in High Schools: An Empirical Study Analysing Teachers’ Opinions

Creating and Evaluating a Visual Programming Course Based on Student Experience

“Why Can’t I Learn Programming?” The Learning and Teaching Environment of Programming

Explore related subjects.

• Artificial Intelligence
• Digital Education and Educational Technology

Aivaloglou, E., Hermans, F. (2016). How kids code and how we know: An exploratory study on the scratch repository. In: Proceedings of the 2016 ACM Conference on International Computing Education Research (pp. 53–61). ACM

Alshaigy, B., Kamal, S., Mitchell, F., Martin, C., Aldea, A. (2015). Pilet: An interactive learning tool to teach python. In: Proceedings of the Workshop in Primary and Secondary Computing Education (pp. 76–79). ACM

Andujar, M., Jimenez, L., Shah, J., & Morreale, P. (2013). Evaluating visual programming environments to teach computing to minority high school students. The Journal of Computing Sciences in Colleges , 29 (2), 140–148.

Google Scholar  

Anniroot, J., de Villiers, M. R. (2012). A study of Alice: A visual environment for teaching object-oriented programming. In: Proceedings of the IADIS International Conference on Information Systems.

Armoni, M., Meerbaum-Salant, O., & Ben-Ari, M. (2015). From scratch to real programming. ACM Transactions on Computing Education (TOCE) , 14 (4), 25.

Ateeq, M., Habib, H., Umer, A., Rehman, M. U. (2014). C++ or python? which one to begin with: A learner’s perspective. In: International Conference on Teaching and Learning in Computing and Engineering (LaTiCE) (pp. 64–69). IEEE

Benson, B. P. (2008). How to meet standards, motivate students, and still enjoy teaching!: four practices that improve student learning. Thousand Oaks: Corwin Press.

Bergin, J., Agarwal, A., & Agarwal, K. (2003). Some deficiencies of c++ in teaching cs1 and cs2. ACM SIGPlan Notices , 38 (6), 9–13.

Article   Google Scholar  

Carl O’Brien, N. I. M. Joe Humphreys (2016). Concern over drop-out rates in computer science courses. http://www.irishtimes.com/news/education/concern-over-drop-out-rates-in-computer-science-courses-1.2491751 , Accessed January 26, 2017

Carlisle, M. C., Wilson, T. A., Humphries, J. W., & Hadfield, S. M. (2005). Raptor: a visual programming environment for teaching algorithmic problem solving. Acm Sigcse Bulletin , 37 (1), 176–180.

Cheung, J. C., Ngai, G., Chan, S. C., & Lau, W. W. (2009). Filling the gap in programming instruction: A text-enhanced graphical programming environment for junior high. ACM SIGCSE Bulletin, ACM , 41 , 276–280.

Cilliers, C., Calitz, A., & Greyling, J. (2005). The effect of integrating an iconic programming notation into cs1. ACM SIGCSE Bulletin, ACM , 37 , 108–112.

Cliburn, D. C. (2008). Student opinions of alice in cs1. In: Frontiers in Education Conference, 38th Annual, IEEE, pp. T3B–1.

Cooper, S. (2010). The design of alice. ACM Transactions on Computing Education (TOCE) , 10 (4), 15.

Daly, T. (2011). Minimizing to maximize: an initial attempt at teaching introductory programming using alice. Journal of Computing Sciences in Colleges , 26 (5), 23–30.

da Silva Ribeiro, R., de Oliveira Brandão, L., Faria, T. V. M., Brandäo, A. A. F. (2014). Programming web-course analysis: how to introduce computer programming? In: IEEE, Frontiers in Education Conference (FIE). IEEE

Davies, S., Polack-Wahl, J. A., Anewalt, K. (2011). A snapshot of current practices in teaching the introductory programming sequence. In: Proceedings of the 42nd ACM technical symposium on Computer science education (pp. 625–630). ACM

Donnelly, K. (2016). Computer Science finally on the way for Leaving Cert students. http://www.independent.ie/irish-news/education/computer-science-finally-on-the-way-for-leaving-cert-students-34576921.html , Accessed March 14, 2017

Dorling, M., White, D. (2015). Scratch: A way to logo and python. In: Proceedings of the 46th ACM Technical Symposium on Computer Science Education, ACM, pp 191–196

Eid, C., & Millham, R. (2012). Which introductory programming approach is most suitable for students: Procedural or visual programming? American Journal of Business Education (Online) , 5 (2), 173.

Federici, S. (2011). A minimal, extensible, drag-and-drop implementation of the c programming language. In: Proceedings of the 2011 conference on Information technology education, pp. 191–196. ACM

Garlick, R., Cankaya, E. C. (2010). Using alice in cs1: A quantitative experiment. In: Proceedings of the fifteenth annual conference on Innovation and technology in computer science education (pp. 165–168). ACM

Giordano, D., Maiorana, F. (2014). Use of cutting edge educational tools for an initial programming course. In: Global Engineering Education Conference (EDUCON), 2014 IEEE (pp. 556–563). IEEE

Grandell, L., Peltomäki, M., Back, R. J., Salakoski, T. (2006). Why complicate things?: introducing programming in high school using python. In: Proceedings of the 8th Australasian Conference on Computing Education (Vol. 52, pp. 71–80). Australian Computer Society, Inc.

Greyling, J., Cilliers, C., Calitz, A. (2006). B#: The development and assessment of an iconic programming tool for novice programmers. In: ITHET’06, 7th International Conference on Information Technology Based Higher Education and Training, pp 367–375). IEEE

Gupta, D. (2004). What is a good first programming language? Crossroads , 10 (4), 7–7.

Hunt, J. M. (2015). Python in cs1-not. Journal of Computing Sciences in Colleges , 31 (2), 172–179.

Ivanović, M., Budimac, Z., Radovanović, M., Savić, M. (2015). Does the choice of the first programming language influence students’ grades? In: Proceedings of the 16th International Conference on Computer Systems and Technologies (pp. 305–312). IEEE

Johnsgard, K., McDonald, J. (2008). Using alice in overview courses to improve success rates in programming i. In: CSEET’08. IEEE 21st Conference on Software Engineering Education and Training (pp. 129–136). IEEE

Keele, S. (2007). Guidelines for performing systematic literature reviews in software engineering. In: Technical report, Ver. 2.3 EBSE Technical Report. EBSE, sn

Koitz, R., Slany, W. (2014). Empirical comparison of visual to hybrid formula manipulation in educational programming languages for teenagers. In: Proceedings of the 5th Workshop on Evaluation and Usability of Programming Languages and Tools (pp 21–30). ACM

Kyfonidis, C., Moumoutzis, N., Christodoulakis, S. (2015). Block-c: A block-based visual environment for supporting the teaching of c programming language to novices.

Laakso, M. J., Kaila, E., Rajala, T., Salakoski, T. (2008). Define and visualize your first programming language. In: ICALT’08, Eighth IEEE International Conference on Advanced Learning Technologies (p. 324). IEEE

Lahtinen, E., Ala-Mutka, K., & Järvinen, H. M. (2005). A study of the difficulties of novice programmers. Acm Sigcse Bulletin, ACM , 37 , 14–18.

Leping, V., Lepp, M., Niitsoo, M., Tõnisson, E., Vene, V., Villems, A. (2009). Python prevails. In: Proceedings of the International Conference on Computer Systems and Technologies and Workshop for PhD Students in Computing (p. 87). ACM

Lewis, C. M. (2010). How programming environment shapes perception, learning and goals: logo vs. scratch. In: Proceedings of the 41st ACM technical symposium on Computer science education (pp. 346–350). ACM

Luxton-Reilly, A. (2016). Learning to program is easy. In: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (pp. 284–289). ACM

Major, L., Kyriacou, T., & Brereton, O. P. (2012). Systematic literature review: Teaching novices programming using robots. IET Software , 6 (6), 502–513.

Maloney, J., Resnick, M., Rusk, N., Silverman, B., & Eastmond, E. (2010). The scratch programming language and environment. ACM Transactions on Computing Education (TOCE) , 10 (4), 16.

Mannila, L., de Raadt, M. (2006). An objective comparison of languages for teaching introductory programming. In: Proceedings of the 6th Baltic Sea conference on Computing education research: Koli Calling (pp. 32–37). ACM

Mannila, L., Peltomäki, M., & Salakoski, T. (2006). What about a simple language? Analyzing the difficulties in learning to program. Computer Science Education , 16 (3), 211–227.

Matsuzawa, Y., Ohata, T., Sugiura, M., Sakai, S. (2015). Language migration in non-cs introductory programming through mutual language translation environment. In: Proceedings of the 46th ACM Technical Symposium on Computer Science Education (pp. 185–190). ACM

Meerbaum-Salant, O., Armoni, M., Ben-Ari, M. (2011). Habits of programming in scratch. In: Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education (pp. 168–172). ACM

Mendeley. (2017). Mendeley Reference Manager. https://www.mendeley.com/ . Accessed March 17, 2017

at the MIT Media Lab LKG. (2017). Scratch Statistics. https://scratch.mit.edu/statistics/ . Accessed March 09, 2017

Montero, S., Díaz, P., Díez, D., Aedo, I. (2010). Dual instructional support materials for introductory object-oriented programming: classes vs. objects. In: IEEE, Education Engineering (EDUCON) (pp. 1929–1934). IEEE

Mullins, P., Whitfield, D., Conlon, M. (2009). Using alice 2.0 as a first language. Journal of Computing Sciences in Colleges, 24 (3), 136–143

NCCA. (2017). Leaving certificate computer science draft curriculum specification. https://www.ncca.ie/media/3184/lc-computerscience.pdf . Accessed January 15, 2018

Nikula, U., Sajaniemi, J., Tedre, M., & Wray, S. (2007). Python and roles of variables in introductory programming: experiences from three educational institutions. Journal of Information Technology Education , 6 , 199–214.

Nolan, K., Bergin, S. (2016). The role of anxiety when learning to program: A systematic review of the literature. In: Proceedings of the 16th Koli Calling International Conference on Computing Education Research (pp. 61–70). ACM

Parker, B. (2011). Teaching experiences with alice for high school students. Journal of Computing Sciences in Colleges , 27 (2), 148–155.

Price, T. W., Barnes, T. (2015). Comparing textual and block interfaces in a novice programming environment. In: Proceedings of the eleventh annual International Conference on International Computing Education Research (pp. 91–99). ACM

Quille, K., Bergin, S., Mooney, A. (2015). Programming: Factors that Influence Success Revisited and Expanded. In: International Conference on Enguaging Pedagogy (ICEP).

Ranade, A. G. (2016). Introductory programming: Let us cut through the clutter! In: Proceedings of the 2016 ACM Conference on Innovation and Technology in Computer Science Education (pp. 278–283). ACM

Robinson, W. (2016). From scratch to patch: Easing the blocks-text transition. In: Proceedings of the 11th Workshop in Primary and Secondary Computing Education (pp. 96–99). ACM

Sandoval-Reyes, S., Galicia-Galicia, P., & Gutierrez-Sanchez, I. (2011). Visual learning environments for computer programming. In: Electronics, Robotics and Automotive Mechanics Conference (CERMA) pp 439–444). IEEE

Seals, C., Mcmillian, Y., Rouse, K., Agarwal, R., Johnson, A. W., Gilbert, J. E., et al. (2008). Computer gaming at every age: A comparative evaluation of alice. i-Manager’s Journal of. Educational Technology , 5 (3), 1.

Slany, W. (2012). Catroid: A mobile visual programming system for children. In: Proceedings of the 11th International Conference on Interaction Design and Children (pp. 300–303). ACM

Smith, N., Sutcliffe, C., Sandvik, L. (2014). Code club: Bringing programming to uk primary schools through scratch. In: Proceedings of the 45th ACM technical symposium on Computer science education (pp. 517–522). ACM

Software T. (2017). TIOBE Index for March 2017. http://www.tiobe.com/tiobe-index/ . Accessed March 15, 2017

Sykes, E. R. (2007). Determining the effectiveness of the 3d alice programming environment at the computer science i level. Journal of Educational Computing Research , 36 (2), 223–244.

Tangney, B., Oldham, E., Conneely, C., Barrett, S., & Lawlor, J. (2010). Pedagogy and processes for a computer programming outreach workshop the bridge to college model. IEEE Transactions on Education , 53 (1), 53–60.

Weintrop, D. (2015). Blocks, text, and the space between: The role of representations in novice programming environments. In: 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) (pp. 301–302). IEEE

Weintrop, D., Wilensky, U. (2015). To block or not to block, that is the question: students’ perceptions of blocks-based programming. In: Proceedings of the 14th International Conference on Interaction Design and Children (pp. 199–208). ACM

Download references

Acknowledgements

This work was assisted through the support of funding received lfrom the John and Pat Hume scholarship, Maynooth University.

Author information

Authors and affiliations.

Department of Computer Science, Maynooth University, Maynooth, Kildare, Ireland

Mark Noone & Aidan Mooney

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Aidan Mooney .

Rights and permissions

Reprints and permissions

About this article

Noone, M., Mooney, A. Visual and textual programming languages: a systematic review of the literature. J. Comput. Educ. 5 , 149–174 (2018). https://doi.org/10.1007/s40692-018-0101-5

Download citation

Received : 20 September 2017

Revised : 05 March 2018

Accepted : 07 March 2018

Published : 15 March 2018

Issue Date : June 2018

DOI : https://doi.org/10.1007/s40692-018-0101-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Programming
• First Programming Language
• Visual Languages
• Textual languages
• Systematic literature review
• Find a journal
• Publish with us
• Track your research

Software engineering and programming languages

Software engineering and programming language researchers at Google study all aspects of the software development process, from the engineers who make software to the languages and tools that they use.

About the team

We are a collection of teams from across the company who study the problems faced by engineers and invent new technologies to solve those problems. Our teams take a variety of approaches to solve these problems, including empirical methods, interviews, surveys, innovative tools, formal models, predictive machine learning modeling, data science, experiments, and mixed-methods research techniques. As our engineers work within the largest code repository in the world, the solutions need to work at scale, across a team of global engineers and over 2 billion lines of code.

We aim to make an impact internally on Google engineers and externally on the larger ecosystem of software engineers around the world.

Team focus summaries

Developer tools.

Google provides its engineers’ with cutting edge developer tools that operate on codebase with billions of lines of code. The tools are designed to provide engineers with a consistent view of the codebase so they can navigate and edit any project. We research and create new, unique developer tools that allow us to get the benefits of such a large codebase, while still retaining a fast development velocity.

Developer Inclusion and Diversity

We aim to understand diversity and inclusion challenges facing software developers and evaluate interventions that move the needle on creating an inclusive and equitable culture for all.

Developer Productivity

We use both qualitative and quantitative methods to study how to make engineers more productive. Google uses the results of these studies to improve both our internal developer tools and processes and our external offerings for developers on GCP and Android.

Program Analysis and Refactoring

We build static and dynamic analysis tools that find and prevent serious bugs from manifesting in both Google’s and third-party code. We also leverage this large-scale analysis infrastructure to refactor Google’s code at scale.

Machine Learning for Code

We apply deep learning to Google’s large, well-curated codebase to automatically write code and repair bugs.

Programming Language Design and Implementation

We design, evaluate, and implement new features for popular programming languages like Java, C++, and Go through their standards’ processes.

Automated Software Testing and Continuous Integration

We design, implement and evaluate tools and frameworks to automate the testing process and integrate tests with the Google-wide continuous integration infrastructure.

Featured publications

Highlighted work.

Some of our locations

Some of our people

Andrew Macvean

• Human-Computer Interaction and Visualization
• Software Systems

Caitlin Sadowski

• Data Management
• Information Retrieval and the Web

Charles Sutton

• Machine Intelligence
• Natural Language Processing
• Software Engineering

Ciera Jaspan

Domagoj Babic

• Algorithms and Theory
• Distributed Systems and Parallel Computing

Emerson Murphy-Hill

Franjo Ivancic

• Security, Privacy and Abuse Prevention

Kathryn S. McKinley

• Hardware and Architecture

Marko Ivanković

Martín Abadi

Hans-Juergen Boehm

Hyrum Wright

Lisa Nguyen Quang Do

Danny Tarlow

Petros Maniatis

• Mobile Systems

Albert Cohen

Kaiyuan Wang

Dustin C Smith

Harini Sampath

Phitchaya Mangpo

We're always looking for more talented, passionate people.

TECHNOLOGY AND CODE article

Students' performances in computer programming of higher education for sustainable development: the effects of a peer-evaluation system.

• 1 School of Arts, Southeast University, Nanjing, China
• 2 Department of Finance, Providence University, Taichung, Taiwan
• 3 Department of Healthcare Administration and Medical Informatics, Kaohsiung Medical University, Kaohsiung, Taiwan
• 4 Department of Fashion Business and Merchandising, Ling Tung University, Taichung, Taiwan
• 5 Department of Automatic Control Engineering, Feng Chia University, Taichung, Taiwan

Modern education attaches great importance to interdisciplinary skills, among which computational thinking is a core element, and heralds a new era. IT application has shaped education in the 21st century. Computational thinking has provided further impetus for building an all-encompassing social network and fostering a DIY culture enabled by digital technologies. One empirical study used four apps to test children's development in computational thinking and fluency. The article will help students overcome their fears of coding. Peer reviews provide students with an opportunity to learn from each other and become more motivated. These reviews also serve as feedback for teachers to evaluate students' performance. Experimental design is used in this study, and a peer review system is implemented. Freshmen attending a programming class in a university are used as samples. At the class, students write computer programs with f-Chart, which provides a graphical user interface for students to learn programming logic and design. Zuvio, a cloud-based interactive response system, is used to conduct the peer reviews. The data of this study are analyzed through R. The results show not only an improvement in students' learning performance but also a gap between students' peer review scores and teachers' evaluation scores. Learning feedback and evaluation is crucial to transform education between students and teachers into a sustainable cycle system.

Introduction

Learning styles and performance assessment have been evolving with time. Traditional teaching strategies put higher educational institutions in a tight spot to engage students' interests and improve performance. Information technology (IT) has become a key driver of global economic growth, especially for sustainable education. Students must cultivate computational thinking by learning computer programming language in order to meet the need for IT development. The capability of computer programming language is one of the basics for IT development. However, students often have fears and difficulties in learning computer programming language. Performance in learning computer programming language logic is studied in this research. The scores of midterm and final examinations are used to evaluate the improvement of students' learning performances. The sustainability of the development of computer programming language will be a future issue.

The four important aspects of learning are learner-centered, knowledge-centered, assessment-centered, and community-centered. “Assessment-centered” could be viewed as the central part in the learning process. If learners and teachers underestimate the assessment, it will have a considerable gap between the instruction and learning.

By contrast, traditional assessment techniques and teacher–student relationship put enormous pressure on students to such an extent that they lose enthusiasm for learning. On the other hand, if instructors use peer review as feedback, it will make teacher–student relationship more productive. The problem in the past was the influence by Asian cultural background because many problems and obstacles in learning were caused from learner's own social pressure. Instruction nowadays should change to student-centered.

The research object of this study is college freshman learners of information science. Whether they are beginners or have background knowledge of the course in high school is taken into account. They were divided into experimental groups to teach by peer review. The self-made results were reported during the semester examination. Peers and teachers are allowed to grade; the control group uses traditional assessment to teach, and test papers are distributed during the semester examinations, so that students can answer and test based on the test questions, and then the teachers review the grades.

The literature review of this study focuses on difficulties in learning programming logic, equation programming, introduction to interactive instruction platforms, and further reading.

Literature Review

To address this challenge, universities have introduced campus-wide programming courses. However, intimated by the English-language user interface and programming language, many students tend to lose confidence, lack motivation to complete the course, etc. ( Calder, 2010 ; Chen et al., 2020 ). Thus, this study employed the block programming interface to improve the learning environment of programming, stimulate students' interest in programming, and improve students' impression that programming is difficult. The teaching model employed in the study encouraged student interaction and discussion, cultivating the learning atmosphere that allows peer discussion and work sharing. This teaching model can also enhance students' abilities in expression, problem-solving, and promoting group cooperation. Through flow chart programming, block programming interface allows students to focus the programming learning on understanding programming logic. At the completion of programming, students can present their works on mobile phones. In addition, this programming approach can also control hardware devices, increasing the richness of the outcome and enhancing students' interest in learning programming. When evaluating students' learning outcomes, this study employed peer evaluation for mutual evaluation, encouraging students to analyze the strengths and weakness of other students' works to have further understanding of their own works. Peer scoring and teacher scoring were used to analyze the research results to examine whether student achievement, learning proficiency, and learning enthusiasm have improved.

Information education is an integral part of modern higher education. It is worth mentioning that this form of education is not to spoon-feed students with programming syntax but to teach them how to think like a programmer and develop creative thinking to solve problems ( Calder, 2010 ; Finn and Garner, 2011 ; Hillsdale et al., 2019 ; Ion et al., 2019 ; Hickey et al., 2020 ). The focus of promoting programming education is to develop and guide students' programming logic ability. In the past, students' learning aims in programming courses were command interface. For students who are new to programming, it is relatively difficult, which caused students to have no interests in programming at all ( Kordaki, 2012 ). In order to arouse students' interest, we must understand students' bottlenecks in learning programming ( Liu et al., 2018a ). After investigation, there are three steps that can arouse students' interests in programming. The first step is that students need to immediately display their self-designed programming works, such as apply communication to control automatic mobile vehicles and robots, experience how to use block program designing, and control designing to solve problems in order to understand the application and fun of programming ( Liu et al., 2018b ). The second step is to simplify the interface of program coding, and the third step is to train students' problem-solving ability ( Luaces et al., 2018 ) to keep students motivated as a sustainable development goal. By applying block program design, students are taught to learn program designing to solve problems and design related information function applications and integrate other areas to create innovative applications. While students work with their peers in the practical application of the topic, their independent thinking abilities are also developed ( Lundstrom and Baker, 2009 ). Instruction means granting students to obtain specific abilities and skills. Therefore, how to make good use of evaluation methods to help learners is the ultimate sustainable development goal of instruction, and it is also a significant issue for educators in higher education to care about ( Mine and Colin, 2018 ).

In recent years, peer review and self-evaluation have been receiving growing attention on how to better motivate students, improve their performance, foster a supportive learning environment, and develop sustainable education ( Murillo-Zamorano and Montanero, 2018 ).

Peer review can be conducted through group discussions to help students forge closer bonds, develop critical thinking, and improve learning efficiency in the process. Past research has found that peer evaluation provides instant feedback among students, which can foster them the ability of solving problems and improve their leaning performance ( Nielsen, 2014 ). In the process of problem-based learning, students will actively seek out problems, realize the core of problems, and then solve problems, that is, using peer evaluation could get rid of the dilemma of the problem-solving. In the peer learning mode, students will advance a conscious attitude to their own learning to promote and deepen what they have learned.

Block Programming

The purpose of information education is to enable students to learn how to solve problems through applying information. Therefore, the purpose of teaching students programming is not to make them memorize programming language but to make them learn how to solve problems by applying programming logic ( Calder, 2010 ; Liu et al., 2018a ). According to extant investigations, there are three keys to inspire students in learning programming ( Kordaki, 2012 ; Hillsdale et al., 2019 ). First, in the process of programming, students can view instantly the output of the programming, for example, using a program to control a robot or an automatic mobile vehicle; students can see immediately the result of the robot being controlled by the program they have just accomplished or modified. Second, the software interface for programming is based on graphic, which makes it easier for students to understand. Third, guiding students' thinking logic in writing programs and allow students to learn how to write a complete program step by step. Compared to the traditional programming language development interface, block programming can better meet the aforementioned three keys. Students can focus on thinking the logic of programming, and it can stimulate students' interests in learning programming. Thus, it is a language more applicable for beginners learning programming ( Piteira and Costa, 2013 ; Luaces et al., 2018 ; Papadakis, 2018 ; Papadakis and Orfanakis, 2018 ; Ion et al., 2019 ; Papadakis and Kalogiannakis, 2019 ; Ladias et al., 2021 , 2022 ).

Self-Assessment

In the process of learning, self-assessment allows students to learn the extent of their own growth. Research has found that student self-assessment is more effective than teacher evaluation to improve student learning outcome ( Kordaki, 2012 ). Nielsen (2014) believed that self-assessment not only helps students build self-confidence and find their own weaknesses but also enhances learning motivations, which bring positive impacts on learning performance ( Liu et al., 2018b ). Self-assessment can guide students develop their own learning strategies to improve learning effectiveness ( Lundstrom and Baker, 2009 ).

The recommendations from the extant studies on applying self-assessment in teaching are as follows ( Liu et al., 2018b ):

(1) Prior to peer assessment, the teacher should inform students the assessment approach and criteria in detail.

(2) Students should have a clear understanding of the purpose of self-assessment. The teacher should advise students on how to complete the self-assessment and then give a score based on students' written reports.

(3) Students should participate in establishing assessment criteria, which would help them understand the criteria and thus conduct peer assessment accordingly.

(4) When assessing students' work, teachers should emphasize on the students' development process and focus on giving positive suggestions, which would trigger students' learning motivation.

(5) Self-assessment is a formative assessment, rather than a summative assessment. Teachers should not over-emphasize the importance of self-assessment scores, instead should focus on the process of assessment, allowing students to self-assess in an honest manner.

(6) Teachers should provide feedback on the self-assessment of students.

(7) The students' self-assessment should include the assessment on both the overall performance and detailed items.

(8) Students' self-assessment can be incorporated with peer feedback, allowing students to know how to correct and improve their works.

Self-assessment allows students to point out the strengths and weaknesses of each other's work, inspire their learning motivations, and understand their own drawbacks ( Mine and Colin, 2018 ).

Peer Evaluation

Through peer evaluation, students could increase their chances of interaction with their peers, understand how to modify their works with the peers' suggestions, and defend their own ideas and answers to the questions, which will significantly enhance students' problem-solving capacity ( Liu et al., 2018b ; Mine and Colin, 2018 ; Murillo-Zamorano and Montanero, 2018 ). The social identity theory is an established theoretical framework founded in psychology that promotes understanding in participation within sciences and engineering, including computer programming ( Roy, 2012 ). However, teachers must plan carefully when setting scoring criteria to avoid reducing students' interest in learning ( Nielsen, 2014 ). When using peer evaluation, students' learning motivation and pressure are from other students, which in turn can reduce the opposition between teachers and students ( Seifert and Feliks, 2019 ). However, when conducting peer evaluation, students should focus on their feedback to other students on the improvement of the works, and not general comments ( Nielsen, 2014 ). Thus, to allow peer evaluation to function properly, Finn and Garner ( To and Panadero, 2019 ) made the following suggestions to teachers:

(1) Explain the content and procedure of feedback for peer assessment: The teacher should plan the scope of assessment and explain how it will be preceded to students. When the content of feedback is not clear, students may not under-stand the feedback content and thus unable to provide adequate feedback to improve the student works. Thus, teachers should assist and guide students to clearly express their feedback content so that students can promote their learning outcome through peer assessment.

(2) Provide students with support and encouragement: The teacher should provide support in the learning and emotional aspects of the feedback content ( Liu et al., 2019 ). In the learning aspect, the teacher should assist students in explaining the content of feedback, allowing the assessed students understand how to modify the work ( Liu et al., 2022 ). In the emotional aspect, students should be encouraged to give positive response toward the peer feedback.

(3) Pay attention to the negative feedback students gave: The peers' negative feedback may lead to the resistance of the assessed students, and they may refuse to modify the work following the feedback ( Liu et al., 2016 ). Thus, the teacher should encourage students to, from the perspective of helping their peers grow, take feedback with a positive attitude.

(4) Examine students' learning problems: Teachers should confirm students' learning status through the feedback from peer assessment and adjust their teaching content accordingly.

(5) Discuss the inconsistent views with students: Students may hold opposite perspectives in their feedback. Teachers should assist students through further discussions and integrate their opinions into a summarized feedback with different perspectives.

(6) Based on the aforementioned suggestions, the role a teacher plays in peer evaluation is to guide students in the student assessment system, providing timely assistance when students encounter difficulties so that peer assessment can improve students' learning outcome.

Interactive IT in Classroom

Effective usage of IT can promote the learning outcome of classroom learning. The other theoretical framework used to guide this study is self-efficacy. Self-efficacy is defined as persons' personal beliefs that they can exhibit behaviors necessary to act or perform in a specific role ( Roy, 2012 ). The IT used in this study is Zuvio. Zuvio allows teachers to ask questions via mobile phones and immediately get students' answers in the class. There are two ways to use Zuvio: first, students can download and install Zuvio app in their mobile phones and answer the teacher's questions, and second, without installing Zuvio app, students can scan the QR code through their mobile phone, go to the webpage, and answer the questions. Students' answers will be automatically summed up, and a chart will be generated to allow teacher to view the correct rate of the students' answers to the questions. In addition, Zuvio also provides the function of peer-to-peer assessment, which allow the export of students' answers. The interactive IT in classrooms can increase students' classroom participation and promote peer interaction to enhance student learning outcome.

Zuvio enables the instructor and students to engage in real-time interaction, which gives simultaneous feedback. Apart from the online application, this software can be integrated with offline courses to meet both instructors' and students' needs, that is, using innovative techniques to facilitate real-time interaction and increase learning effectiveness. To be more specific, Zuvio has four procedures: the cloud test prepared by the instructor before class; the virtual test taken by students; visualized interaction; and quantitative analysis. Real-time checks of students' progress and performance evaluation are examined through anonymous or non-anonymous questions which are tracked by Zuvio for analysis.

R can enhance its statistic or drawing functions through suites, for example, Big Data Suite, a suite that networks with other programming languages ( Zahedi et al., 2021 ). In developing R-related apps, R Studio is the most popular integrated development environment (IDE). R Studio can turn data into charts and turn figures into visual graphics, allowing data analyzers to more efficiently understand the meaning of the figures.

The aforementioned literature reveals that the traditional teaching method poses a big challenge to students who have no background knowledge of programming language. Gradually, they lose self-confidence and motivation to complete the course. Learner-friendly solutions have been provided. However, they aim to incentivize students, rather than motivating them to move forward. Although such a method does not spoon-feed students, their learning performance is undermined. Considering that language syntax is the only practice-oriented programming course, it is all the more important to remove learning hurdles and teach students how to think critically and creatively. In fact, this is the fundamental aim of programming education.

Learner-friendly elements to foster computational thinking include natural language description of blocks, drag-and-drop interactions, and modular languages. Compared to traditional, text-based programming, this course is more intellectually demanding, yet it enables learners to have a good grasp of programming logic. Prior research has shown that a new visually enhanced programming environment is more welcoming, thus reducing cognitive load. That explains why a visually enhanced programming environment with a drag-and-drop interface is the most popular tool among beginners. Block equation programming can increase learning effectiveness. An interactive instruction platform is another major plus. Classcraft is clear evidence that technology has great potential for engaging students. Existing research has shown that students' input and attitudes have a positive influence on learning outcomes.

This research proves that the innovative teaching method that combines peer assessment, block-based programming, and interactive instruction platform does increase programming learning outcomes. Thus, the teaching method designed by this research has 2-fold benefits: improving learning performance and developing college-wide programming education.

Conceptual Framework

The conceptual framework of the study is shown in Figure 1 . It begins with how to plan and write the content and teaching materials for block programming, followed by the instruction of the first five modules, while dividing the class into the peer assessment group (experiment group) and traditional instruction group (control group) to explore whether different instruction models affect students' performance. Next, the students' achievement and assignment-related data will be compared; the teaching review and reflection will be conducted for the second half of the semester to adjust the instruction model accordingly. Finally, the analysis on students' learning outcome will be performed by using the adjusted instruction model.

Figure 1 . Conceptual framework.

Figure 1 shows the structure and procedures of this study. It took 10 weeks to develop a module-based course schedule, whose first five units are covered in this research. The following 9 weeks were devoted to the instruction and learning of these five units. The traditional teaching method was applied to this process. An experiment had been run for 2 weeks to compare results reported by the peer assessment group and the controlled one. Both groups were given a test in the same content, but in different orders of questions; 1 week was spent examining the scoring difference between peer assessment and instructor evaluation, followed by reflection and feedback. This led to adjustments to the course schedule and teaching methods for the next program. At the same time, discussions were conducted on how different teaching methods influence homework and achievements. Another 3 weeks were devoted to comparing grades and homework scores. Statistical analysis was conducted to explore the possibility of adjusting the teaching method for the five units before the conclusion of student achievements was drawn.

Brigade Laboratory Method

It is imperative to understand how to adjust the teaching progress and the content during the course with the students' feedback. In order to understand the learning status of students, assessment is the most direct and specific way in education. It is often used to know whether students have absorbed the content of the classroom. The study targeted the information science-related majors in a basic programming course; the subjects are divided into the experiment group and control group, as shown in Table 1 . The experiment group adopted the new teaching model of peer assessment and self-evaluation; the control group adopted the traditional lecture model. After the midterm examination, the first teaching review was conducted, and the curriculum was adjusted accordingly to be followed by the second half-semester of teaching. It was expected that a new teaching model will bring a better learning outcome of programming than the traditional one. The students under new teaching model should outperform the students under traditional teaching model in applying programming in problem-solving and logical thinking.

Table 1 . Experiment group and control group for research hypotheses.

Implementation Steps and Peer Evaluation Rubrics

The subjects of the study were the freshman of information science, and the course was programming. The study employed peer evaluation to calculate the students' grade, allowing students to discover the strength and weakness of self and each other's works to elevate their programming capacity. There were six stages in the evaluation:

(1) Turning in assignment: Students turned in their assignment to the digital learning platform system.

(2) Assignment evaluation: Students scored the assignment by following assessment criteria.

(3) Teacher evaluation: Teacher scored the assignment.

(4) Peer assessment: Students scored the students making the presentation.

(5) Self-assessment: Students also scored their own works while scoring others.

(6) Assignment performance: Upon the completion of peer and teacher assessment, students made corrections based on the feedback.

The traditional evaluation method is divided into two stages:

(1) Examination stage: The teacher issues midterm and final examinations in the classroom.

(2) Teacher rating: Teachers rate students' midterm and final examinations.

This study built the peer evaluation rubrics ( Table 2 ) based on the following principles:

(1) Design the scoring criteria for the course of programming logic.

(2) Build the assessment aspects and define the assessment standard.

(3) The description of the assessment should be easy to understand.

(4) The items and subheadings of assessment standard are clear and precise.

(5) Determine the standards of five levels, for example, excellent, very good, and fair.

(6) Determine the score range of each level.

(7) Calculate the number of operations using the criteria example of the assessment.

(8) Listen to the student feedback and correct accordingly.

(9) Communicate and discuss the new assessment criteria with students and ensure they understand the assessment standard.

Table 2 . Peer assessment rubric.

The virtual test is taken via a smartphone or computer whose system is preset with a student roster and test-taking tutorial. When logging on to the cloud testing system, students will receive a notification about the course and its related test. This real-time system allows examinees to remain anonymous, interact with peers, and reflect on lessons learned from the test. As a result, the learners find the course more appealing. It is worth mentioning that before the test, students must familiarize themselves with peer assessment rubrics and Zuvio manual ( Table 2 ).

Data Processing and Analysis Approach

The study analyzed the distribution of data from the experiment group and control group against teacher's evaluation while making score analysis on the major and non-major groups. To further understand the students' learning progress and the scoring differences between the student and teacher, the study calculated the following two indicators and analyzed the changes in the indicators.

This study examines the performance of 72 students. Before grouping, they answered 10 questions to test foundational knowledge about programming logic. Among the examinees, 38 had some background knowledge, while the rest 34 had none. The former scored 6.78 points, and the latter 3.56, a difference of 3.22. This suggests that background information does matter.

Student Professional Approach Level

Through student professional approach-level indicator ( Table 3 ), one can understand the scoring difference between the student and teacher. The teacher can discuss with students the difference and let students understand the teaching objectives the teacher values when conducting scoring in class. The figure in Table 3 is the percentage growth rate. The range of figure refers to Liu (2016) ; if it is 1+, it means that it is approaching 100% with the teacher; 0.95 means that the student reaches about 95% of the teacher's professional degree; 0.8 means that the student achieves about 80% of the teacher's professionalism. This figure is the same as the teacher's rating of 0 ~ 100. Observe the teacher and students' rating from a third-party perspective to obtain an objective professional value.

Table 3 . Student professional approach table.

Student Self-Professional Growth Change

After students experienced assessment more than once, they can conduct comparison on the degree of difference based on the score data and understand their learning status according to the magnitude of the difference. Students can discuss whether the benchmark of scoring results can effectively improve the learning outcome and understand the accomplishment status of current curriculum and teaching objectives based on multiple records. Formula 2 can be used to calculate student professional growth change value, which can be applied to Table 4 to understand the student professional growth rate.

Table 4 . Student self-professional growth change table.

Analytical Assessment on Course Content

The five stages of the course content in this study are detailed in Table 5 . Stage 1 is course planning. The English program code of the text interface is first changed into block programming or a set of block programming material is designed to allow easy learning for students. Stage 2 is to incorporate peer assessment into the course when it was appropriate. This is the learning stage for students using the development environment provided by Mit App Inventor 2. The reason for adopting Inventor 2 is that students can have easy access with a Google account. In addition, it is operated on the webpage, and no additional program download is required. Inventor 2 is a block-type programming, which is in line with the research direction of the study. Another purpose of this stage is to employ the self-developed material to observe whether students have improved their performance in programming through using block program. Stage 3 is the assessment stage. The tool employed is Zuvio. Zuvio was developed to strengthen students' and teachers' development capacity. The many functions it provides include peer assessment, teacher–student interaction, and student feedback. Zuvio also provides quantitative analysis; thus, this study mainly takes Zuvio to conduct peer assessment and student feedback. Stage 4 is data analysis. Software used is R; the reasons are described in sections Literature Review, Conceptual Framework, and Learning Effectiveness Analysis. In this stage, mainly the assessment data are used for analysis and integration to analyze students' learning status. The last stage is improvement which improves the course content based on the analytical data and consolidated data, including student feedback.

Table 5 . Student self-professional growth change.

Learning Effectiveness Analysis

The subjects of the study were the students in the mobile device development course. A total of 72 students divided into 18 groups participated in this research, and the students were grouped freely, with each team having four students. Experimental class (A) and control class (B) were conducted in this research. Experimental class included non-information major students (A1) and information major students (A2), and control class included non-information major students (B1) and information major students (B2). The analysis of students' learning effectiveness is given as follows.

Relationship Between the Professional Level of the Teacher and Student Scoring

The scores of the midterm and final examinations were calculated by using formula 1, and the results are shown in Table 6 . It can be seen that both the degree of approach between the midterm and final examinations was >0.9, indicating the difference between the students' and teacher's understanding of the scoring method is small. According to Table 7 , the progress range of class A was 0.02, indicating the students have made slight improvement in grades, but there still existed room for improvement; the progress range of class B was 0.19, indicating students have made significant growth in the progress range.

Table 6 . Teacher and student score approaching degree.

Table 7 . Student self-grow change.

Figure 2 shows two sets of midterm and final examination scores for experimental group A and control group B. The scores are based on peer assessment and teacher's evaluation. Another score indicating cognitive differences between the two groups for midterm examination performance hovers around 0.93 and 0.94, suggesting the influence of unfamiliarity with the rubrics. By the final examination, cognitive difference of group B is narrowed down to 0.90. However, students in this group make no notable progress. By contrast, average scores in experimental group A are higher than those of midterm examination grades, while the cognitive difference stays at 0.94, showing learning progress and familiarity with the rubrics.

Figure 2 . Teacher and student score approaching degree.

Approaching Relationship Between Student and Teacher Scores

Teacher scores and student scores are presented in the distribution chart; the horizontal axis is the teacher score, and the vertical axis is the student score. Through the peer evaluation methods at the middle and end of the period, the range of the distribution points of the results is observed to understand the students' learning effectiveness. The progress rate is calculated by the aforementioned formula. Figures 3 , 4 , respectively, show the midterm and final examination grade distribution of the two classes, where the small point is the average score of the 18 groups, and the larger point is the overall average of the group. According to the original classification of the non-information major group (A1), the experimental class information major group (A2), the control class non-information major group (B1), and the experimental class information major group (B2), the average score of the information major group is higher than that of the non-information major group, and the scores of the experimental class are higher than those of the control class. It can be seen that when the experimental class uses the peer evaluation method, the students' learning status and results are better than the those using the traditional evaluation method of the control class.

Figure 3 . Midterm score approach: classes A1, A2, B1, and B2.

Figure 4 . Final score approach: classes A1, A2, B1, and B2.

Midterm examination results show the following findings: the average score of A2 and B2 students who have some knowledge of information science is higher than that of A1 and B1 peers who are first-time learners within the same class; the grade gap among learners in A2 and B2 is narrower, indicating that background knowledge of programming logic does make a difference.

Final examination results show a similar trend in average scores. Nevertheless, the grade gap among all students becomes narrower, with the average scores of the four groups increasing. It is clear proof that such a teaching method is well-received by students and that peer pressure has positive impacts. In other words, such innovative programming and peer assessment can enhance student performance.

Student scores and teacher scores were presented, respectively, in the distribution chart, with the horizontal axis as the student seat number and vertical axis as the scores given by the student and teacher. There were two types of graphs in the charts representing teacher or student scores, respectively. The round dot represents the score from peer assessment; the triangle represents the score from the teacher with the score indicated on the left. The figure below the graph represents the approaching degree between student and teacher scores, as is shown in Figures 5 , 6 . There were two classes in this study. The student numbers of the class are 46 for class A and 26 for class B. Each class was divided into several groups, with 11 groups in class A and 7 groups in class B. According to Figures and formula 1, it can be observed that the scores of both teacher and students in most groups were >1, indicating the students and teacher had the same understanding of the assessment criteria. A few groups did show a significant deviation from the teacher scores, for example, for group 7 of class A, the value calculated by the formula was only 0.217, which indicated that a significant gap existed in the judging score benchmark of the group and the teacher.

Figure 5 . Group professional-level variation: class A.

Figure 6 . Group professional-level variation : class B.

The assessment also allowed students to substitute their assignment scores into formula 2 to measure their own progress. Students' self-evaluation rubrics are shown in Figures 7 , 8 , with the horizontal axis as the student seat number and vertical axis as the scores given by the students and teacher. The round dot represents the score of the first assignment; the triangle represents the score of the second assignment. The scores from both assignments are shown at the left of the distribution chart, and the value below indicate the range of progress or regression calculated by using formula 2. The final self-assessment results showed most students demonstrated slight degree of positive growth. For example, No. 7 of class B made the highest growth rate of 0.42 in the self-assessment, progressing from 43 points to 74 points, as 0.42 > 0.4 representing substantial growth in accomplishing the assignment. The score of No. 22 ( Figures 7 , 8 ) among group 7 ( Figures 3 , 6 ) in the control class is given for peer evaluation at the end of the period, but the report was not uploaded to the Zuvio platform. According to the regulations, the teacher cannot score it, so it is 0 point.

Figure 7 . Self-assessment: class A.

Figure 8 . Self-assessment: class B.

The study is fully aware that students who have no background knowledge of programming find it challenging to familiarize themselves with the English-language interface and syntax. To address this challenge, this research introduces an innovative teaching method that is reinforced by peer assessment rubrics and interactive instruction platforms and explores the potential of peer pressure and block programming. Through discussion and sharing, students become more adept at articulating ideas, solving problems, and cooperating with team members.

The study introduces an innovative teaching method that is reinforced by peer assessment rubrics and interactive instruction platforms, and explores the potential of peer pressure and block programming. Grades of experimental and control groups are analyzed, indicating that such a method does improve programming learning performance in the traditional educational context. Thus, the study suggests that college teachers and programming trainers adopt such a method to turn peer pressure into enhanced performance.

It is not surprising to find that students who have no prior exposure to programming do not fare well in the traditional educational context, which in turn undermines their motivation and confidence. Therefore, introducing more learning strategies and platforms should not aim to engage students' interest but to motivate them to overcome difficulties and improve learning outcomes.

Figure 9 shows cognitive differences among A1, A2, B1, and B2 for midterm and final examination scores. Average scores based on peer assessment and teacher's evaluation rise to some extent. Cognitive differences for B1 and B2 are higher than those of A1 and A2, suggesting the former two groups' unfamiliarity with the rubrics. Between A1 and A2, with the former knowing nothing about programming and the latter having some background knowledge, cognitive differences are smaller, which can be attributed to the familiarity with peer assessment rubrics.

Figure 9 . Midterm and final examination score comparison: classes A1, A2, B1, and B2.

The experimentation results indicate that grouping is not a differentiating factor because both groups showed signs of improvement. Nevertheless, it is worth noting that when engaging in peer assessment at the initial stage, students with no background information were more productive than those taught by the traditional method, suggesting that peer pressure is a motivator and enhancer of learning outcomes. In the second stage, the learners also delivered better performance than the control group, suggesting that peer pressure, companionship, and instruction do improve beginners' performance.

The instruction of computational thinking begins with its definition and teaching method. The first step is to differentiate between mathematical thinking and computational thinking. The latter covers four aspects: data application, modeling and simulation, problem-solving, and systemic thinking. This research focuses on the latter two. To this end, the authors examine related studies, as well as 101 block-based programming environments, of which 46 are analyzed to compare different designs that support the transition to text-based programming. Totally, 10 principles of tool construction are provided. An innovative teaching method is created to facilitate the learning of block-based programming and to improve learning performance.

The study analyzes the data obtained from the procedures, and peer assessment is a distinguishing feature of it. Class A and B students reported that high-school or vocational school courses related to information science did influence their performance. However, this perceived difference is not validated by either peer assessment or instructor evaluation. Both scores are higher than 0.90, suggesting that students and the instructor have a similar understanding of the course's objectives. One possible explanation is that the familiarization with the peer assessment rubrics gives students a deeper understanding of what they are expected to achieve. As for the indicator of self-improvement, the difference among class A students between midterm and final examination results is 0.02, suggesting considerable room for growth. The difference for class B students is 0.19, indicating significant progress. Therefore, the familiarization with the peer assessment rubrics has an influence on the scoring distribution of instructor evaluation and peer review. The dispersion for the midterm examination is more scattered. In the final examination, the dispersion is more clustered, indicating that benchmark values between peer assessment and instructor evaluation are close. It also shows that students and their peers achieved a general consensus on how to gain the specialized knowledge and how to gauge peer assessment.

The study incorporated peer evaluation mechanism in a classroom setting to understand whether the new instruction model was beneficial to student learning in higher education. Exploring role identity within the social identity theory enables us to understand better how the role or identity impacts engagement, retention, and persistence. The study employed Zuvio for peer evaluation. The results showed that in classes A and B, for those who majored in information science in regular high school or vocational high school, the variation in professional approach degrees was limited. Thus, in terms of programming, the variation between student scores and teacher scores was limited. The students' professional approach degree value was higher than 0.90, which indicated that students and teacher had similar understanding of the teaching objectives. On the other hand, when analyzing the self-professional growth changes of classes A and B, it was found that the self-professional growth change range for class A was 0.02, which indicated that there was still room for improvement in learning growth. Class B demonstrated a 0.19 change range. In other words, all students have sustainable development in their reflective learning. In learning, there was a significant progress in the final than in the midterm examination. In the approaching relationship between the teacher and students, both teacher scores and student scores were presented in the distribution chart. Through the dot distribution, it can be found that the spread was more scattered, indicating a larger variation between teacher and student scores. In final assessment, the spread was more intense, indicating students' peer assessments already have a similar perspective with the teacher assessment, and they shared the same understanding on professional knowledge and assessment methods.

The results show that peer evaluation reaches a sustainable development goal in programming education. Non-information major students within peer evaluation have higher effective learning in programming than those in traditional teaching methods in the beginning stage, which means that students are more likely to learn programming because of peer pressure. Non-information major students also show higher learning effectiveness in the second stage than those in the control class, which means that under the pressure and guidance of peers, peer assessment teaching makes students more attractive to programming courses.

The study designed an innovative teaching method whose effectiveness is examined by analyzing the grades of experimental and control groups, with a greater emphasis on the learning performance of the former. Students in this group have no prior exposure to programming. Nevertheless, final examination results indicate that their performance is improved, better than that of their counterparts in the control group. In other words, the innovative teaching method does make a difference.

Given that programming logic and similar courses are attached greater importance by higher education institutions, how to replace the traditional teaching method with innovative ones and ensure the latter suits the current learning environment deserves serious attention. The methodology and innovative teaching method designed by this research can provide some inspiration for college teachers and programming trainers who can guide students to cope with peer pressure, and improve learning performance.

Limitation and Future Work

The study has some limitations: (a) the course spans only one semester; (b) few classes and learners engage in the experiment; and (c) the division of category or group needs to be more precise. Thus, statistical accuracy needs to be increased. If this experiment is run for years to have more data collected, statistical accuracy will be ensured, and clear patterns be found.

Based on the conclusion, the authors suggest that peer assessment, an innovative teaching method, should be adopted for module-based course instruction. Compared to the traditional teaching style, peer pressure and comparison motivate students to do better and engage their learning interest in programming. But the one-size-fits-all approach should be avoided. The peer assessment rubrics should take into account learners' characteristics. The fundamental criterion should be whether such assessment maximizes learning outcomes.

Data Availability Statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding authors.

Ethics Statement

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the [patients/participants OR patients/participants legal guardian/next of kin] was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author Contributions

T-CH and Y-HC: conceptualization and data curation. T-LC: methodology. T-LC and C-YC: software. T-CH, Y-HC, and C-CC: validation. T-CH: formal analysis, investigation, and funding acquisition. T-CH and C-CC: resources, supervision, and project administration. T-LC and T-CH: writing—original draft preparation, writing—review and editing, and visualization. All authors have read and agreed to the published version of the manuscript.

This research was supported by the Fundamental Research Funds for the Central Universities (Grant Number 3213049408), projects under MOST 108-2221-E-390-005 and MOST 108-2622-E-390-002-CC3. This research was also supported by the major education and teaching reform projects in Fujian undergraduate colleges and universities in 2019 (grant FBJG20190284).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Calder, N. (2010). Using scratch: an integrated problem-solving approach to mathematical thinking. Aust. Prim. Math. Classr . 15, 9–14.

Google Scholar

Chen, T. L., Hsiao, T. C., Kang, T. C., Wu, T. Y., and Chen, C. C. (2020). Learning programming language in higher education for sustainable development: point-earning bidding method. Sustainability . 12, 1–14. doi: 10.3390/su12114489

CrossRef Full Text | Google Scholar

Finn, G. M., and Garner, J. (2011). Twelve tips for implementing a successful peer assessment, medical Teacher 33, 443–446. doi: 10.3109/0142159X.2010.546909

PubMed Abstract | CrossRef Full Text | Google Scholar

Hickey, D. T., Robinson, J., Fiorini, S., and Feng, Y. A. (2020). Internet-based alternatives for equitable preparation, access, and success in gateway courses. Internet High. Educ . 44, 1–9. doi: 10.1016/j.iheduc.2019.100693

Hillsdale, N. J., Lawrence, E., Piteira, M., and Costa, C. (2019). “Learning Computer Programming: Study of Difficulties in Learning Programming,” in International Conference on Information Systems and Design of Communication , ACM Lisbon, Portugal (2013).

Ion, G., Marti, A.S., and Morell, I. A. (2019). Giving or receiving feedback: which is more beneficial to students' learning? Assess. Eval. High. Educ. 44, 124–138. doi: 10.1080/02602938.2018.1484881

Kordaki, M. (2012). Diverse categories of programming learning activities could be performed within scratch. Procedia. Soc. Behav. Sci . 46, 1162–1166. doi: 10.1016/j.sbspro.2012.05.267

Ladias, A., Karvounidis, T., and Ladias, D. (2021). Classification of the programming styles in scratch using the SOLO taxonomy. Adv. Mob. Learn. Educ. Res . 1, 114–123. doi: 10.25082/AMLER.2021.02.006

Ladias, A., Karvounidis, T., and Ladias, D. (2022). Forms of communications in scratch and the SOLO taxonomy. Adv. Mob. Learn. Educ. R . 2, 234–245. doi: 10.25082/AMLER.2022.01.007

Liu, J., Guo, X. Y., Gao, R. Q., Fram, P., Ling, Y., Zhang, H., et al. (2018a). Students' learning outcomes and peer rating accuracy in compulsory and voluntary online peer assessment. Assess. Eval. High. Educ . 44, 835–847. doi: 10.1080/02602938.2018.1542659

Liu, X. Y., Li, L., and Zhang, Z. H. (2018b). Small group discussion as a key component in online assessment training for enhanced student learning in web-based peer assessment. Assess. Eval. High. Educ . 43, 207–222. doi: 10.1080/02602938.2017.1324018

Liu, Y. C. (2016). Development and evaluation of rubrics for assessing the performance of college students in human resource management courses. Educ. J. NHCUE . 33, 77–108. doi: 10.3966/199679772016063301003

Liu, Z., Kong, X., Liu, S., Yang, Z., and Zhang, C. (2022). Looking at MOOC discussion data to uncover the relationship between discussion pacings, learners' cognitive presence and learning achievements. Educ. Inf. Technol . doi: 10.1007/s10639-022-10943-7

Liu, Z., Liu, S., Liu, L., Sun, J., Peng, X., and Wang, T. (2016). Sentiment recognition of online course reviews using multi-swarm optimization-based selected features. Neurocomputing . 185, 11–20. doi: 10.1016/j.neucom.2015.12.036

Liu, Z., Yang, C., Sylvio, R, Liu, S., and Wang, T. (2019). Temporal emotion-aspect modeling for discovering what students are concerned about in online course forums. Interact. Learn. Environ . 27. 598–627. doi: 10.1080/10494820.2019.1610449

Luaces, O., Diez, J., and Bahamonde, A. (2018). A peer assessment method to provide feedback, consistent grading and reduce students' burden in massive teaching settings. Comput. Educ . 126, 283–295. doi: 10.1016/j.compedu.2018.07.016

Lundstrom, K., and Baker, W. (2009). To give is beter than to receive: the benefits of peer review to the reviewer's own writing. J. Second Lang. Writ . 18, 30–43. doi: 10.1016/j.jslw.2008.06.002

Mine, C. R., and Colin, R. (2018). Infrastructure and tools for teaching computing throughout the statistical curriculum. Am. Sta t. 72, 58–65. doi: 10.1080/00031305.2017.1397549

Murillo-Zamorano, L. R., and Montanero, M. (2018). Oral presentations in higher education: a comparison of the impact of peer and teacher feedback. Assess. Eval. High. Educ . 43, 138–150. doi: 10.1080/02602938.2017.1303032

Nielsen, K. (2014). Self-assessment methods in writing instruction: a conceptual framework, successful practices and essential strategies. J. Res. Read . 37, 1–16. doi: 10.1111/j.1467-9817.2012.01533.x

Papadakis, S. (2018). Is pair programming more effective than solo programming for secondary education novice programmers?: a case study. Int. J. Web-Based Learn. Teach. Technol. 13, 1–16. doi: 10.4018/IJWLTT.2018010101

Papadakis, S., and Kalogiannakis, M. (2019). Evaluating the effectiveness of a game-based learning approach in modifying students' behavioural outcomes and competence, in an introductory programming course. A case study in Greece. Int. J. Teach. Case Stud. 10, 235–250. doi: 10.1504/IJTCS.2019.102760

Papadakis, S., and Orfanakis, V. (2018). Comparing novice programing environments for use in secondary education: app inventor for android vs. alice. Int. J. Technol. Enhanc. Learn . 10, 44–72. doi: 10.1504/IJTEL.2018.088333

Piteira, M., and Costa, C. (2013). “Learning computer programming: study of difficulties in learning programming,” in Proceedings of the 2013 International Conference on Information Systems and Design of Communication (Lisbon), 75–80.

Roy, K. (2012). “App inventor for android: report from a summer camp,” in SIGCSE '12 Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (Raleigh, NC), 283–288.

Seifert, T., and Feliks, O. (2019). Online Self-Assessment and Peer-Assessment as a Tool to Enhance Student-Teachers' Assessment Skills. Assessment & Evaluation in Higher Education . 44, 169–185. doi: 10.1080/02602938.2018.1487023

To, J., and Panadero, E. (2019). Peer assessment effects on the self-assessment process of first-year undergraduates. Assess Eval. High. Educ . 44, 920–932. doi: 10.1080/02602938.2018.1548559

Zahedi, L., Batten, J., Ross, M., Potvin, G., Damas, S., Clarke, P., et al. (2021). Gamification in education: a mixed-methods study of gender on computer science students' academic performance and identity development. J. Comput. High. Educ . 33, 441–474. doi: 10.1007/s12528-021-09271-5

Keywords: technical education, sustainable education, peer review, graphical user interface, block-based computer programming language logic design

Citation: Hsiao T-C, Chuang Y-H, Chen T-L, Chang C-Y and Chen C-C (2022) Students' Performances in Computer Programming of Higher Education for Sustainable Development: The Effects of a Peer-Evaluation System. Front. Psychol. 13:911417. doi: 10.3389/fpsyg.2022.911417

Received: 04 April 2022; Accepted: 20 June 2022; Published: 13 September 2022.

Reviewed by:

Copyright © 2022 Hsiao, Chuang, Chen, Chang and Chen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Tzer-Long Chen, tlchen@kmu.edu.tw ; Chih-Cheng Chen, chenccheng@fcu.edu.tw

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

Subject-Oriented Pages

Other related pages.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Journal Proposal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Harnessing the power of large language models for automated code generation and verification.

1. Introduction

1.1. problem statement, 1.2. research objectives, 2. background, 2.1. finite state machines in robotics, 2.2. large language models (llms), 2.3. guaranteeing the safety and quality of automatically generated code, 2.4. proposed approach, 2.4.1. human-programmed and carefully curated “software blocks”, 2.4.2. use of a second llm to verify results from a first one, 3. architecture, 3.1. prepare context information, 3.2. code generation—step 1: automatically generates a plan from the specification provided by the user, 3.3. plan validation: evaluate plan and used elements.

• Actions should follow a certain logical order (for example: an object has to be picked up before it is put somewhere, something has to be opened before it can be closed, ...).
• Actions do not overflow the robot’s capabilities (for example: picking up more than one element at the same time).
• A validation to check that the robot’s actions will create a final state of the system that matches the final state indicated by the user (for example: validating that ‘all door should be closed’).

3.4. Code Generation—Step 2: Convert the Plan into a Sequence of Allowed Actions (“Skills” That the Robot Can Execute)

•  Step 1) Move to the fridge location.
•  Step 2) Open the fridge.
•  Step 3) Pick up the egg.
•  Step 4) Close the fridge.

4. Experimental Setup and Validation

• A synthetic setup based on the iTHOR environment [ 33 ]. iTHOR is an environment within the AI2-THOR framework, which includes a set of interactive objects and more than 120 scenes and provides accurate modeling of the physics of the world. The AI2-THOR framework is a popular simulated environment for embodied AI research, providing a set of realistic and interactive scenarios for testing and training AI systems.
• A physical setup based on a dual-arm robot that performs pick-and-place operations on a factory assembly line.

4.1. Synthetic Setup

• Translating high-level action commands from the Flexbotics system into low-level motor control commands that can be executed in the iTHOR environment.
• Converting sensor data from the AI2-THOR environment into a format that can be processed by the Flexbotics system.
• Handling communication between the Flexbotics system and the iTHOR environment, ensuring that actions are executed correctly and that the system receives accurate feedback.
• Fridge: Is an appliance that is initially closed. Initially, it contains an egg and a head of lettuce.
• Egg: a pickable food located initially inside the fridge.
• Lettuce: a pickable food located initially inside the fridge.
• Stove: is an appliance for cooking.
• StoveBurner: the burner of the stove appliance.
• Pan: a frying pan is a tool used for cooking.
• Plate: an empty plate container located on top of the countertop.
• Countertop: there are two countertops: one aligned with the fridge and a kitchen island countertop.
• Basin: a kitchen sink.
• Toaster: Is an appliance on the countertop. Can be switched on and off.
• Knife: is a slicing tool inside a drawer.
• Tomato: located on the kitchen island countertop.
• Bread: a bread loaf on the kitchen island countertop.
• Drawer: a drawer that contains a knife.
• Environment and elements in it: which elements can be found in the synthetic setup around the robot (the ones in the simulated kitchen shown above) and their initial state.
• Allowed robot actions (basic actions that can later be translated to Flexbotics skills): – moving around. – picking up and using tools. – act on elements (to switch on/off, open/close, slice food, etc.). – moving elements around.
• Translation of robot actions into Flexbotics skills: describing how to generate a valid JSON from a high-level plan.

Synthetic Setup: Experimental Results

• USER REQUEST 1 : “Put the tomato in the fridge. Make sure that in the end the fridge is closed.”
• Total number of tokens used by Generator LLM: 1047 .
• S1: a junior user with no prior experience generating JSON files.
• S2: a junior user with some previous experience generating JSON files.
• S3: a senior programmer with experience generating JSON files.
• S4: a senior with experience programming robots and generating JSON files.
• USER REQUEST 2 : “I want the robot to slice some tomato and then put it on a plate.”
• Total number of tokens used by Generator LLM: 1107 .
• USER REQUEST 3 : “Prepare a toast of sliced bread, and put it on the plate with a slice of tomato on top of it. ”
• Total number of tokens used by Generator LLM: 1427 .

4.2. Physical Robot

• A dual-arm Kawada Nextage robot
• A stereo camera.
• A set of vitro-ceramic electric cooker components that must be assembled by the robot.

4.3. Results after Validation on Synthetic and Physical Setups

• User Request: the mission requested by the user.
• Actions in JSON: the number of sequential actions created in the generated JSON.
• Total Tokens: the total number of tokens evaluated by the Generator LLM in the first step of code generation (as a measure of the complexity of the user request).
• Verification Loops: the number of times the Supervisory LLM requested a change in the plan.
• Successful Simulation: indicates if the mission was fully executed in the simulation and the goals indicated by the user were achieved.

4.4. Scalability

5. conclusions and future work, author contributions, data availability statement, conflicts of interest.

• IFR Secretariat Blog—AI in Robotics. 2022. Available online: https://ifr.org/post/ai-in-robotics-blog (accessed on 4 September 2024).
• Ogheneovo, E. On the Relationship between Software Complexity and Maintenance Costs. J. Comput. Commun. 2014 , 2 , 51631. [ Google Scholar ] [ CrossRef ]
• Ben-Ari, M. Finite state machines. In Elements of Robotics ; Springer: Berlin/Heidelberg, Germany, 2018; pp. 55–61. [ Google Scholar ] [ CrossRef ]
• Balogh, R.; Obdržálek, D. Using Finite State Machines in Introductory Robotics: Methods and Applications for Teaching and Learning. In Using Finite State Machines in Introductory Robotics: Methods and Applications for Teaching and Learning ; Springer: Berlin/Heidelberg, Germany, 2019; pp. 85–91. [ Google Scholar ] [ CrossRef ]
• Zhou, H.; Min, H.; Lin, Y.; Zhang, S. A Robot Architecture of Hierarchical Finite State Machine for Autonomous Mobile Manipulator. In A Robot Architecture of Hierarchical Finite State Machine for Autonomous Mobile Manipulator ; Springer: Berlin/Heidelberg, Germany, 2017; pp. 425–436. [ Google Scholar ] [ CrossRef ]
• Foukarakis, M.; Leonidis, A.; Antona, M.; Stephanidis, C. Combining Finite State Machine and Decision-Making Tools for Adaptable Robot Behavior. In Proceedings of the Universal Access in Human-Computer Interaction—Aging and Assistive Environments: 8th International Conference, UAHCI 2014, Heraklion, Crete, Greece, 22–27 June 2014. [ Google Scholar ]
• Herrero, H.; Outón, J.L.; Puerto, M.; Sallé, D.; López de Ipiña, K. Enhanced Flexibility and Reusability through State Machine-Based Architectures for Multisensor Intelligent Robotics. Sensors 2017 , 17 , 1249. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Herrero, H.; Outón, J.L.; Esnaola, U.; Sallé, D.; López de Ipiña, K. Development and evaluation of a Skill Based Architecture for applied industrial robotics. In Proceedings of the 2015 4th International Work Conference on Bioinspired Intelligence (IWOBI), San Sebastian, Spain, 10–12 June 2015; pp. 191–196. [ Google Scholar ] [ CrossRef ]
• Diab, M.; Pomarlan, M.; Beßler, D.; Akbari, A.; Rosell, J.; Bateman, J.; Beetz, M. SkillMaN—A skill-based robotic manipulation framework based on perception and reasoning. Robot. Auton. Syst. 2020 , 134 , 103653. [ Google Scholar ] [ CrossRef ]
• Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.; Foote, T.; Leibs, J.; Wheeler, R.; Ng, A. ROS: An Open-Source Robot Operating System. In Proceedings of the ICRA Workshop on Open Source Software, Kobe, Japan, 12–17 May 2009; Volume 3. [ Google Scholar ]
• Zhao, W.X.; Zhou, K.; Li, J.; Tang, T.; Wang, X.; Hou, Y.; Min, Y.; Zhang, B.; Zhang, J.; Dong, Z.; et al. A Survey of Large Language Models. arXiv 2023 , arXiv:2303.18223. [ Google Scholar ] [ CrossRef ]
• META. Meta-LLama-3-70B-Instruct. 2024. Available online: https://huggingface.co/meta-llama/Meta-Llama-3-70B (accessed on 4 September 2024).
• DEEPINFRA. Meta-LLama-3-70B-Instruct. 2024. Available online: https://deepinfra.com/meta-llama/Meta-Llama-3-70B-Instruct (accessed on 4 September 2024).
• Naveed, H.; Khan, A.U.; Qiu, S.; Saqib, M.; Anwar, S.; Usman, M.; Akhtar, N.; Barnes, N.; Mian, A. A Comprehensive Overview of Large Language Models. arXiv 2024 , arXiv:2307.06435. [ Google Scholar ] [ CrossRef ]
• Will Henshall. TIME: 4 Charts That Show Why AI Progress Is Unlikely to Slow Down. 2023. Available online: https://time.com/6300942/ai-progress-charts/ (accessed on 4 September 2024).
• Schäfer, M.; Nadi, S.; Eghbali, A.; Tip, F. An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation. arXiv 2023 , arXiv:2302.06527. [ Google Scholar ] [ CrossRef ]
• D’Silva, V.; Kroening, D.; Weissenbacher, G. A Survey of Automated Techniques for Formal Software Verification. Trans. Comp.-Aided Des. Integ. Cir. Sys. 2008 , 27 , 1165–1178. [ Google Scholar ] [ CrossRef ]
• Floridi, L.; Chiriatti, M. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds Mach. 2020 , 30 , 681–694. [ Google Scholar ] [ CrossRef ]
• Ye, W.; Ou, M.; Li, T.; Chen, Y.; Ma, X.; Yanggong, Y.; Wu, S.; Fu, J.; Chen, G.; Wang, H.; et al. Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility. arXiv 2023 , arXiv:2305.10235. [ Google Scholar ] [ CrossRef ]
• Bisk, Y.; Zellers, R.; Bras, R.L.; Gao, J.; Choi, Y. PIQA: Reasoning about Physical Commonsense in Natural Language. arXiv 2019 , arXiv:1911.11641. [ Google Scholar ] [ CrossRef ]
• Brown, T.B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. Language Models are Few-Shot Learners. arXiv 2020 , arXiv:2005.14165. [ Google Scholar ] [ CrossRef ]
• Kocoń, J.; Cichecki, I.; Kaszyca, O.; Kochanek, M.; Szydło, D.; Baran, J.; Bielaniewicz, J.; Gruza, M.; Janz, A.; Kanclerz, K.; et al. ChatGPT: Jack of all trades, master of none. Inf. Fusion 2023 , 99 , 101861. [ Google Scholar ] [ CrossRef ]
• Yang, X.; Li, Y.; Zhang, X.; Chen, H.; Cheng, W. Exploring the Limits ChatGPT for Query or Aspect-based Text Summarization. arXiv 2023 , arXiv:2302.08081. [ Google Scholar ] [ CrossRef ]
• De Wynter, A.; Wang, X.; Sokolov, A.; Gu, Q.; Chen, S.Q. An evaluation on large language model outputs: Discourse and memorization. Nat. Lang. Process. J. 2023 , 4 , 100024. [ Google Scholar ] [ CrossRef ]
• Delobelle, P.; Tokpo, E.; Calders, T.; Berendt, B. Measuring Fairness with Biased Rulers: A Comparative Study on Bias Metrics for Pre-trained Language Models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, WA, USA, 15 July 2022; Carpuat, M., de Marneffe, M.C., Meza Ruiz, I.V., Eds.; Association for Computational Linguistics: Stroudsburg, PA, USA, 2022; pp. 1693–1706. [ Google Scholar ] [ CrossRef ]
• Domnich, A.; Anbarjafari, G. Responsible AI: Gender bias assessment in emotion recognition. arXiv 2021 , arXiv:2103.11436. [ Google Scholar ] [ CrossRef ]
• ESA. Guide to Software Verification and Validation ; ESA PSS-05-10 Issue 1, Revision 1, March 1995; ESA: Paris, France, 1995. [ Google Scholar ]
• Peleska, J.; Haxthausen, A.; Schnieder, E.; Tarnai, G. Object Code Verification for Safety-Critical Railway Control Systems. In Proceedings of the Formal Methods for Automation and Safety in Railway and Automotive Systems (FORMS/FORMAT 2007), Braunschweig, Germany, 25–26 January 2007; pp. 184–199. [ Google Scholar ]
• Iyyer, M.; Wieting, J.; Gimpel, K.; Zettlemoyer, L. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. arXiv 2018 , arXiv:1804.06059. [ Google Scholar ] [ CrossRef ]
• Alzantot, M.; Sharma, Y.; Elgohary, A.; Ho, B.J.; Srivastava, M.; Chang, K.W. Generating Natural Language Adversarial Examples. arXiv 2018 , arXiv:1804.07998. [ Google Scholar ] [ CrossRef ]
• Gao, J.; Lanchantin, J.; Soffa, M.L.; Qi, Y. Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifiers. arXiv 2018 , arXiv:1801.04354. [ Google Scholar ] [ CrossRef ]
• Zhao, Z.; Dua, D.; Singh, S. Generating Natural Adversarial Examples. arXiv 2018 , arXiv:1710.11342. [ Google Scholar ] [ CrossRef ]
• Kolve, E.; Mottaghi, R.; Han, W.; VanderBilt, E.; Weihs, L.; Herrasti, A.; Deitke, M.; Ehsani, K.; Gordon, D.; Zhu, Y.; et al. AI2-THOR: An Interactive 3D Environment for Visual AI. arXiv 2022 , arXiv:1712.05474. [ Google Scholar ] [ CrossRef ]
• Zhang, H.; Li, L.H.; Meng, T.; Chang, K.W.; den Broeck, G.V. On the Paradox of Learning to Reason from Data. arXiv 2022 , arXiv:2205.11502. [ Google Scholar ] [ CrossRef ]
• Wu, Z.; Qiu, L.; Ross, A.; Akyürek, E.; Chen, B.; Wang, B.; Kim, N.; Andreas, J.; Kim, Y. Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks. arXiv 2024 , arXiv:2307.02477. [ Google Scholar ] [ CrossRef ]
• Wu, Z.; Bai, H.; Zhang, A.; Gu, J.; Vydiswaran, V.V.; Jaitly, N.; Zhang, Y. Divide-or-Conquer? Which Part Should You Distill Your LLM? arXiv 2024 , arXiv:2402.15000. [ Google Scholar ] [ CrossRef ]
• Plaat, A.; Wong, A.; Verberne, S.; Broekens, J.; van Stein, N.; Back, T. Reasoning with Large Language Models, a Survey. arXiv 2024 , arXiv:2407.11511. [ Google Scholar ] [ CrossRef ]
• Huang, J.; Chang, K.C.C. Towards Reasoning in Large Language Models: A Survey. arXiv 2023 , arXiv:2212.10403. [ Google Scholar ] [ CrossRef ]
• Qiao, S.; Ou, Y.; Zhang, N.; Chen, X.; Yao, Y.; Deng, S.; Tan, C.; Huang, F.; Chen, H. Reasoning with Language Model Prompting: A Survey. arXiv 2023 , arXiv:2212.09597. [ Google Scholar ] [ CrossRef ]
• Levy, M.; Jacoby, A.; Goldberg, Y. Same Task, More Tokens: The Impact of Input Length on the Reasoning Performance of Large Language Models. arXiv 2024 , arXiv:2402.14848. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Antero, U.; Blanco, F.; Oñativia, J.; Sallé, D.; Sierra, B. Harnessing the Power of Large Language Models for Automated Code Generation and Verification. Robotics 2024 , 13 , 137. https://doi.org/10.3390/robotics13090137

Antero U, Blanco F, Oñativia J, Sallé D, Sierra B. Harnessing the Power of Large Language Models for Automated Code Generation and Verification. Robotics . 2024; 13(9):137. https://doi.org/10.3390/robotics13090137

Antero, Unai, Francisco Blanco, Jon Oñativia, Damien Sallé, and Basilio Sierra. 2024. "Harnessing the Power of Large Language Models for Automated Code Generation and Verification" Robotics 13, no. 9: 137. https://doi.org/10.3390/robotics13090137

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

An Evaluation Framework and Comparative Analysis of the Widely Used First Programming Languages

Affiliations Department of Computer Science, Abdul Wali Khan University, Mardan, Pakistan, Faculty of Information Technology, University of Central Punjab, Lahore, Pakistan

Affiliation Department of Computer Science, Abdul Wali Khan University, Mardan, Pakistan

Affiliation Faculty of Information Technology, University of Central Punjab, Lahore, Pakistan

Affiliation Department of Mathematics, Abdul Wali Khan University, Mardan, Pakistan

* E-mail: [email protected]

• Muhammad Shoaib Farooq, 
• Sher Afzal Khan, 
• Farooq Ahmad, 
• Saeed Islam, 

• Published: February 24, 2014
• https://doi.org/10.1371/journal.pone.0088941
• Reader Comments

Computer programming is the core of computer science curriculum. Several programming languages have been used to teach the first course in computer programming, and such languages are referred to as first programming language (FPL). The pool of programming languages has been evolving with the development of new languages, and from this pool different languages have been used as FPL at different times. Though the selection of an appropriate FPL is very important, yet it has been a controversial issue in the presence of many choices. Many efforts have been made for designing a good FPL, however, there is no ample way to evaluate and compare the existing languages so as to find the most suitable FPL. In this article, we have proposed a framework to evaluate the existing imperative, and object oriented languages for their suitability as an appropriate FPL. Furthermore, based on the proposed framework we have devised a customizable scoring function to compute a quantitative suitability score for a language, which reflects its conformance to the proposed framework. Lastly, we have also evaluated the conformance of the widely used FPLs to the proposed framework, and have also computed their suitability scores.

Citation: Farooq MS, Khan SA, Ahmad F, Islam S, Abid A (2014) An Evaluation Framework and Comparative Analysis of the Widely Used First Programming Languages. PLoS ONE 9(2): e88941. https://doi.org/10.1371/journal.pone.0088941

Editor: Hussein Suleman, University of Cape Town, South Africa

Received: September 18, 2013; Accepted: January 14, 2014; Published: February 24, 2014

Copyright: © 2014 Farooq et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Funding: The authors have no support or funding to report.

Competing interests: The authors have declared that no competing interests exist.

Introduction

Computer programming holds a central importance in the computing curricula. The selection of a programming language for an introductory course of computer programming has always been pivotal as well as contentious [1] , such a language is generally referred to as First Programming Language (FPL). Purpose of the first course in computer programming is to provide conceptual knowledge to the beginners for “understanding the fundamental programming constructs” in such a way that they should be able to program a given problem [2] , [3] . The literature survey reveals [78] , [79] , [80] , [81] that many different programming languages have been used as FPL. During the 1990s, Professor Richard Reid of Michigan State University has been maintaining a list of the languages used as FPL by various different universities and institutes [4] . Later on, this list has been updated till 2006 by Frances Van Scoy [79] , and lately another version of this list has been compiled [78] . A summarized list of number of universities using a particular language as FPL at different times has been presented in Table 1 , which has been compiled by getting data from [4] , [78] , [79] , [80] . It clearly reflects that Pascal remained dominant FPL for a whole decade (1990s) [78] , while Ada and Modula-2 remained consistent during this time. C++ gained popularity in late 90s, whereas Java and Python started to appear in the counts in late 90s. Java emerged as the most widely used FPL beyond 2006, whereas, C++ remained the runner-up throughout this time.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0088941.t001

The genealogy of the programming languages has been presented in Figure 1 . The languages in solid boxes are popular FPLs. The figure clearly shows that newer languages are influenced by some existing languages, which enforces a new language to carry some legacy features of its ancestors. Thus, the size of the new language increases, which in turn poses serious problems in terms of its suitability as an FPL. On the other hand some languages [82] have been designed purely from educational perspective, but they altogether miss out the industrial demands, and hence are not warmly welcomed by the community. This demands a comprehensive evaluation criterion for evaluating the suitability of a language as an appropriate FPL. Many people have presented different sets of requirements [5] , [6] , [7] , [8] for an appropriate FPL. However, these approaches discuss the problem at higher abstraction levels, and to our knowledge, there is no concrete and well defined method for the evaluation of an appropriate FPL.

https://doi.org/10.1371/journal.pone.0088941.g001

The major focus of this article is to figure out a possible way to evaluate the suitability of a language as an FPL. To this end, we have proposed a framework for the evaluation of an FPL which is mainly based on technical and environmental features. The novelty of this work is that while evaluating the languages we have not only relied on relevant research literature, but we have also strongly involved the general programming language rules to evaluate most of the features. We have also defined a scoring function based on the parameters in the defined framework. This scoring function is customizable and can be tuned to the user's preferences. The other contribution of this work is that we have evaluated and ranked widely used FPLs using our proposed framework. Therefore, we have considered different imperative and object oriented programming languages which have been top ranked FPLs at a certain stage. It is evident from Table 1 that Ada, C, C++, Fortran, Java, Modula-2, and Pascal have been the most frequently used FPLs, whereas, Python and C# which have recently gained popularity as FPL. The survey revealed that Scheme is also another reasonably used FPL; however, we do not consider it in our comparison as it is not an imperative language.

The rest of the paper is organized as follows: following the Introduction section, we present the related work. The proposed framework has been discussed in detail in the section “Proposed Framework and Comparative Analysis of Commonly used FPLs” , where we have not only presented the evaluation criterion for each feature, but we have also rated the considered FPLs over it. The scoring function along with the suitability analysis for the programming languages has been presented in the section “Scoring Function” . Finally, we present the conclusion and future directions of this research work.

Related Work

Formal evaluation efforts for the assessment of programming languages are few and far between, and most evidence gathered is anecdotal in nature. Some approaches have been proposed to evaluate the quantitative suitability score for an FPL, for instance, Parker et al. [5] compiled a list of criteria for introductory programming courses at universities. However, this criterion has not been discussed with sufficient technical details of the involved measures, which can be useful for evaluation and scoring purposes.

Clarke [8] used questionnaires to evaluate a programming language. He has demonstrated that a questionnaire involving the cognitive dimensions can be a useful and valuable tool for evaluating the usability of a programming language. But, focusing just on cognitive aspects does not allow comprehensive evaluation or assessment from all aspects.

Gupta [6] discussed requirements for programming languages for beginners, which effectively is a requirement analysis for an appropriate FPL. However, there is no formal assessment mechanism devised for the evaluation of a language for its suitability as an introductory FPL. Similarly, some articles [13] , [14] , [17] present language independent evaluation based on intrinsic and extrinsic criterion for suitability of introductory programming language. Intrinsic criterion is related to language technical aspects such as type safety, syntax, visual vs. textual, compiled vs. interpreted. Extrinsic criterion is related external factors (student demand, industry trend), accessibility (supporting material, text books) and introductory programming course (design, thinking, algorithm social skills). However, these criterion and relevant parameters have been discussed on a surface level and need to be probed further so as to actually evaluate the languages.

McIver [7] proposed a method for comparative evaluation based on the usability of programming language. The interaction of programmer with similar language IDEs was recorded and analyzed for all types of errors made by programmer. The proposed approach by McIver evaluates languages together with similar IDEs; however, it strongly focuses on IDE and undermines the other features. Kölling [42] claims that several tools to support and improve the learning and teaching of programming have been developed, used and researched for many years, but still the problem persists.

Another dimension of research in this area is the comparative evaluation of languages that are widely being used as FPL. For instance, a comparison of Modula-2, Fortran-77, Pascal and C is presented in [10] . Phipps [11] compared C++ and Java from the viewpoint of defects, bugs and productivity rates. Similarly, Hadjerrouit examined Java's suitability as an FPL [12] . A comparison of Ada95, C, C++, and Java with their conformance to the requirements of “Steelman” has been presented in [45] . Another dimension of work, presented in [12] [15] , [16] , [17] by motivated faculty members, is about their dissatisfaction on a language's usability, especially, C++ and Java. This has paved way for other newer languages like Python. These articles present the evaluation of these languages based on their teaching experiences. Another recent but orthogonal dimension of work is to gather the real data about the behavior of the novice programmers [44] .

The above discussion reveals that many efforts have been carried out to evaluate and compare different FPLs, yet no adequate way to assess and compare FPLs exists. This gives rise to the question of the availability of a comprehensive method to evaluate a language's strength as an appropriate FPL, which in turn, helps in comparing the suitability of different languages as FPL. In this work, we focus on defining a comprehensive evaluation criterion for the assessment of a proper FPL, with all relevant and in-depth details. The novelty of this work is that apart from defining the evaluation parameters, we have also presented the related characteristics to evaluate each parameter, and unlike existing approaches our method strongly incorporates the general programming language rules for this purpose. This effectively helps in performing comprehensive evaluation of a language, as well as may be used to compare the suitability of different languages as an appropriate FPL. We have also assigned scores to the widely used FPLs using our framework. Furthermore, we have also devised a score aggregation function so as to quantify and rank the FPLs based on the given criterion.

Proposed Framework and Comparative Analysis of Commonly Used FPls

In this section we present our proposed framework for the suitability analysis of an FPL. Furthermore, the suitability analysis of popular FPLs, based on the parameters defined in this framework, has also been presented. To this end, we analyze each language and assign a qualitative score based on its conformance to each factor related to a parameter. Our proposed framework comprises of two main categories which include technical and environmental feature sets. The technical feature set covers the language theoretical aspects, whereas, the environmental feature set helps evaluating the external factors. These factors have been presented in Table 2 . The feature sets in this framework not only help in evaluating the suitability of an FPL, but also include comprehensive guideline for designing a new FPL.

https://doi.org/10.1371/journal.pone.0088941.t002

Firstly, we have discussed each technical feature separately, which are then followed by the discussion on environmental features, individually. We discuss each feature using the following ingredients: (i) define a feature; (ii) discuss its suitability for the evaluation of an FPL; (iii) define its evaluation criterion; and (iv) evaluate each language in Table 1 to reflect its conformance to each relevant measurable sub features for that feature.

Each feature is further comprised of a few sub features, and while evaluating each language against a feature, we rate it against each defining sub-feature. To this end, we use the following four simple qualitative values: (i) Fully Supported ; (ii) Mostly Supported; (iii) Partially Supported; (iv) Not Supported. We assign an entry of “Fully Supported” to a language against a sub feature if its major implementations generally meet the requirements, whereas, “Not Supported” indicates that requirements are generally not met. The intermediate entry “Partially Supported” shows that some requirements are met, but a major portion of the requirements are not met, while “Mostly Supported” specifies that the requirement is generally met, but some specific requirements are not met. Such qualitative measures have already been used in literature for the language evaluation [45] .

In order to define and evaluate each feature, we have utilized the references available in the literature; statistics related to the languages; sources considered to be language's defining documents; and the implementation of these features in the widely used compilers of that language, essentially with similar semantics. The list of documents considered for this study has been presented in Appendix S1 .

Technical Features

In this section we discuss each technical feature in detail. These technical features have been evaluated by considering a language's conformance to their defining sub-features. Furthermore, these features have also been rated against the aforementioned four qualitative values.

High Level.

A good FPL should not have constructs that concern machine internals and possess no semantic value [18] . Jobs that can easily be managed by compiler or underlying platform should not be privileged to programmer [19] . IBM defined the level of a language as the number of basic assembly language statements it would take to produce the functionality of one statement in the target language [20] . In any high level language one instruction should be equal to three or more assembly language instructions. Table 3 describes levels of popular leading FPLs in which except C all languages are high level [20] . In our evaluation process we have incorporated IBM's criteria. For that reason C is a middle level language.

https://doi.org/10.1371/journal.pone.0088941.t003

It is important to note that by definition it is evaluated in quantitative terms, therefore, we do not map it to the above mentioned qualitative values. However, we treat it in a different manner, as discussed in the scoring function section, where we compute the overall score of a language.

Orthogonality.

Orthogonality means all language constructs follow consistent rules [6] , [9] , [21] , [57] , [58] . As an example, in an orthogonal language keywords cannot be declared as an identifier; and semantics of statement should be predictable. Therefore, an orthogonal language offers the novice programmers a smoother and simpler learning curve. Hence, in order to evaluate the orthogonality of a language we evaluate it based on the following parameters: i) all keywords should be reserved; ii) consistent rules should be applied; and iii) interaction of the constructs should be predictable.

In terms of the mainstream FPLs, all keywords are reserved in Ada, C++, C, Java, Python, Modula-2 and Pascal. However, in Fortran [21] keywords are not reserved and can be declared as an identifier. This in turn, creates serious readability problem as shown in Figure 2(C ode Listing 1). C# provides two types of keywords, i.e. reserved and contextual , the reserved keywords can be declared as an identifier with ‘@’ prefix, while contextual keywords are special words for compiler in certain context and can be declared as an identifier outside the context.

https://doi.org/10.1371/journal.pone.0088941.g002

Consistent rules means that the features of a language are independent of the context of its appearance in a program. If syntactic construct is allowed for one data type, it should be allowed for all the data types available in the language, e.g. in C an array cannot be returned from function, but it is possible to return an array when it is placed in a structure [36] , [58] . Parameter passing rule in Java is orthogonal, primitives are passed by value , whereas objects are passed by reference . All Python and Java objects are created in heap memory. In C++, objects can be created in heap as well as in stack using different syntax. The methods of objects created in stack are accessed through (.) dot operator, whereas (→) arrow operator is used to access members for the objects stored in heap, which is a violation of orthogonality as illustrated in Figure 2 (lines 4, 6 -Code Listing 2).Similarly, in Java and C#, the rule for the initialization of variables is not orthogonal. By default, all class scope primitive variables are implicitly initialized with its default value as illustrated in Figure 3 (line 2 - Code listing 3), and objects are initialized with null . However, the local variables are not initialized by default, and should be explicitly initialized by the programmer as illustrated in Figure 3 (lines 4,6 - Code Listing 3). However, unlike the other languages the variable initialization rule in Ada is orthogonal, as it implicitly assigns a default initial value for all types of variables and also supports in , out and in - out mode semantics for parameter passing.

https://doi.org/10.1371/journal.pone.0088941.g003

Another consideration about orthogonatlity is that it enforces a predictable interaction among the constructs of a language, which means that the compiler writer is bound to implement a certain language construct as prescribed by the language documentation, and the behavior of the implementation should be documented, and needs to be consistent [45] . The defining documents for the languages C and C++ include a very large number of undefined semantics. C++ is constantly evolving and its compilers are mostly complete with respect to only a few versions, and some programs in C/C++ exhibit different behavior on different compilers [45] . Ada exhibits the same problems as some compilers have not completed their transition to newer specifications, although the validation process of Ada (including ACVC test suite) helps to ensure that its compilers implement the entire language [59] . Similarly, Fortran has also been facing the same inconsistencies as it has also been evolving over the years, and thus losing support from many compiler versions. Table 4 shows the evaluation of the considered programming languages in terms of their conformance to the property of orthogonality.

https://doi.org/10.1371/journal.pone.0088941.t004

Strongly Typed.

Strongly typed means all type checking issues are resolved either at compile time, or at run time [21] [22] , [57] . It ensures that no unexpected results occur at runtime due to type mismatching. Thus, it must be checked by the compiler, or by the runtime system, and no automatic conversions should be allowed. The only possible way for type conversion is explicit type casting by the programmer. Strongly typed languages are more reliable, and are easy to program and debug by novices. The concept of strongly type is usually implemented in two forms: dynamic strongly typed, and static strongly typed.

In dynamic strongly typed languages the variables are implicitly declared and the type binding takes place at run time. Variables are independent of type but value has type. The variable type is determined when a value is assigned to a variable using an assignment statement, variables are references defined in stack , and value is an object defined in heap . Type of variable can be changed from one type to another type at run time as illustrated in Figure 4 (lines 2, 5 -Code Listing 4). This certainly increases programming flexibility, but at the same time, decreases early error detection. Python, JavaScript, and Ruby support dynamic strongly typed concept. In Python, incompatible types on the right side of an assignment operator are not detected as errors, rather the type of the left side is simply changed to the incorrect type. For example in Figure4 (line 1, 4 -Code Listing 5) the variables x and y store integer values, and z is storing a list. One needs the assignment statement y = x but accidently assign list z to x variable as illustrated in Figure 4 (line 4 – Code Listing 5). In this case no error is detected, interpreter simply converts variable x to list type. This type of error is hardly detectable by novices, and full errors diagnostics depend on heavy unit testing, which is not possible for novice programmer.

https://doi.org/10.1371/journal.pone.0088941.g004

In static strongly typed languages the variables are explicitly declared and the type binding takes place at compile time. Similarly, all errors related to type are detected at compile time. The type of a variable cannot be changed after its declaration. Fortran, Ada, C/C++, Pascal, Modula-2, Java, and C# belong to static strongly typed category. Fortran supports both explicit and implicit declarations. If a variable is not explicitly declared by programmer, then it is implicitly declared according to following convention: identifier whose name starts with I,J,K,L,M,N or their lower case versions, is implicitly declared to be an Integer; and otherwise is declared to be real.

Ada allows the programmer to defer type checking for a specified type conversion using function Unchecked_Conversion . C/C++ are mostly static strongly typed languages, the only problem is that, the union construct cannot be type checked. Fortran uses e quivalence for union , and the union construct is not type checked. Hence, such type of union is called free union as shown in Figure 5 (line 7 - Code Listing 6) [21] . A special type of union called discriminant union is secure for type checking. Pascal and Ada support this type of union as shown in Figure 5 (line 3- Code Listing 7) [21] . In Ada, Pascal, and Modula-2 union is type safe and more reliable. Java and C# do not support union due to the concerns of type safety.

https://doi.org/10.1371/journal.pone.0088941.g005

Another type conversion issue is boxing and unboxing. Boxing is a process of converting primitive data types to object types, and vice versa is called unboxing [60] , [61] . This augments a language's capability towards static type checking. C# and Java both support this concept. In C# primitive data types are stored onto the stack, and object types are stored in heap. Boxing implicitly converts stack value types to heap objects, and unboxing explicitly converts heap object to stack value as shown in Figure 6 (line 2,3 Code Listing 8). In Java boxing implicitly converts value of primitive types in corresponding object wrapper type. It will not generate some type errors. e.g. % and + =  operators are not available for Java's wrapper Integer type, and the compiler compiles the code without any error as shown in Figure 6 (line 4,5 Code Listing 9). Unboxing to Null object is a reliability issue as it will generate NullPointerException in Java. The usage of boxing concept is not recommended for arithmetic expressions; the safest way is to use this concept for the storage of primitives in collection. Java and C# boxing concept is similar to that of Python for memory allocation in stack and heap. For novices, boxing and unboxing create simplified application of heap and stack, but some types of errors are hardly detectable by the novices. In general, static strongly typed languages help the novice programmers as they help diagnosing all type checking errors earlier at compile time. Table 5 shows the evaluation of the considered programming language as a strongly typed language.

https://doi.org/10.1371/journal.pone.0088941.g006

https://doi.org/10.1371/journal.pone.0088941.t005

Enforceability of Good Habits.

A good FPL should enforce programmers to write clean and consistent code. Good program writing style is based on clarity and readability, and these habits should be encouraged from the beginning [23] , [24] . A good language should not allow: [9] , [62]

• Coercion with demotion (narrowing).
• Expression side effects
• Intermixing of arithmetic, logical and relational operators in Boolean expressions
• Unconventional operator usage and overloading
• Scope overriding

Coercion with demotion (narrowing conversion) results into the loss of data during the processing of mix-mode arithmetic expression. Fortran, Modula-2, Pascal, C++ allow promotion as well as demotion using coercion. Demotion creates data loss problem as shown in Figure 7 (line 6 - Code Listing 10). Ada allows restricted form of mix - mode expressions. C#, Java, and Python discourage the concept of coercion with demotion. In terms of teaching, it is very hard for a teacher to explain coercion at early stages.

https://doi.org/10.1371/journal.pone.0088941.g007

Arithmetic expressions are prone to get affected from functional side effects. Side effect occurs when a function changes a non-local variable or a two-way parameter [21] , [62] . A good programming language produces referentially transparent programs that are more readable as shown in Figure 7(C ode Listing 11). The variables answer1 and answer2 will be equal if function fun has no side effects, whereas, in case of any side effect these variables may not be equal.

There is no concept of function side effects in mathematics, and it is also true in functional programming languages. A good programming language should not violate primitive mathematical rules. e.g. fun (2) + fun(2) is equivalent as 2 * fun(2) in mathematics. Ada supports this by using only in-mode formal parameters. Static local variables produce functional side effects, and Java language does not allow such type of variables. Similarly, a misuse of global variables violates this concept as shown in Figure 8 (line 14,18 - Code Listing 12). Fortran, C, C++, Modula2, Pascal, and Python are prone to such side effects. While, due to object oriented features C++ tends to receive less use of global variables. Java and C# do not support global variables but are still prone to functional side effects. Java and C# do not support global variables and are prone to functional side effects.

https://doi.org/10.1371/journal.pone.0088941.g008

Mixing of arithmetic operators with logical and relational operators creates readability problem as shown in Figure 9 (lines 3,4 - Code Listing 13). It results into frustrating experiences for novices, and debugging of the code becomes tedious if arithmetic operators are used as Boolean expressions. In Mathematics, Boolean algebra AND , OR operators have same precedence, Ada supports this concept. Fortran, Modula-2, C, C++, C#, Java, and Python have different precedence for AND and OR operators. For example, in C arithmetic expressions can be written as Boolean expressions in if statement. In C, scalar variables (numeric or character) and constants can also be used as Boolean expressions, where ‘0’ means false and non-zero is considered as true . A good language should avoid these issues [36] .

https://doi.org/10.1371/journal.pone.0088941.g009

Unconventional operator usage in language design will also create readability problem. For Example, division operator (/) in most of the languages is used for integers as well as real numbers as shown in Figure 9 (line 3 - Code Listing 14). In mathematics (/) operator means real division. In assignment statement double result = first/second; both operands in division are integer type so integer division truncates fractional result. Destination variable result is double so integer result is coerced to double. Here implicit type conversion (coercion) will not be responsible for the data loss. Pascal, Modula-2 and Python provide separate operators for integer and real division. Pascal and Modula-2 use div for integers and ‘/’ for real numbers, whereas Python uses “//” for integer and ‘/’ for real . Other languages use ‘/’ as overloaded operator for both integer and real division. In short, a good FPL should not violate core mathematical rules.

Unconventional operator overloading also create ambiguities. Languages which support abstract data types like Ada, C++, C#, Python, and Fortran95 allow programmer to overload operators. When used for conventional purpose it will enhance readability but unconventional use of operator overloading will create poor readability. For example use of + operator to compare two stack type objects. In order to avoid unconventional operator overloading, Java does not support this feature. Sometimes overloading of AND and OR operators overrides the default behavior of short circuiting in boolean expression, that is why C# and Python do not allow overloading of these operators. Ada defines “and then” and “or else” as short circuit form and these forms cannot be overloaded as shown in Figure 10 (line 1, 3 -Code Listing 16).

https://doi.org/10.1371/journal.pone.0088941.g010

Scope overriding decreases the readability of a program. In most block scope languages, variable name should be unique with in single block, but nested blocks can declare same name variable as parent block. C++ provide scope resolution operator (::) for accessing overridden global variables. In order to avoid scope overriding problem variable names should be unique within single as well as nested scopes. For novice programmer identical names in nested blocks are too error prone and difficult to debug as shown in Figure 10 (line 2, 4 -Code Listing 15). Here Loop condition depends on outer i declared at line 2 but inner i declared at line 4 overrides outer i . Therefore, outer i cannot be accessible within the body of while loop, which results into an infinite loop. C# and Java does not allow scope overriding. A good programming language and coding standards should not allow scope overriding, as it is error prone, especially, for novice programmers [21] , [46] . Table 6 shows the evaluation of the considered languages based on their enforceability of good habits.

https://doi.org/10.1371/journal.pone.0088941.t006

In order to evaluate a language for its strength in security we propose the following parameters: i) language should avoid dangling references; ii) there should not be any memory leakage; (iii) control over array index out of bound; (iv) pointers arithmetic; (v) prevent stack and heap overflows. Every programming language should support controlled aliasing. Aliasing can be brutal to the security of the program [30] , and a programming language is considered to be less secure due to uncontrolled aliasing. Uncontrolled aliasing is a major threat as it may create problems like dangling referencing and memory leakage. Programs that have the keyword new without a matching delete , creates these types of problems. Best practice is to add new keyword and remove delete keyword from a language. Particularly, from an FPL's point of view inclusion of new and delete affects the pedagogical activities both in terms of teaching, and learning. Thus, the provision of automatic garbage collection should be made available in good a programming language.

Dynamic memory management mechanism of Pascal and Modula-2 poses problems like memory leakage as shown in Figure 11 (line 9 - Code Listing 17), and dangling references as shown in Figure 11 (line 8 - Code Listing 18).

https://doi.org/10.1371/journal.pone.0088941.g011

In Java, the keyword new is allowed but there is no delete keyword. Garbage collector is responsible for deleting all non-referenced memory locations in Java, C# and Python. C++ also suffers from memory leakage and dangling reference problems since programmer has to explicitly revoke the memory using the delete keyword. Ada partially elevated dangling reference problem and has no solution for garbage.

Out of bound access in array is another security problem. C/C++ suffer from this problem which leads to read and write operations to unwanted memory locations. Novice programmers can make such mistakes as shown in Figure 12 (line 4, 5 -Code Listing 19). However, Java and C# do not pose such issues and throw array index out-of-bounds exceptions at run time, or report error at compile time. Python, Modula-2, and Ada also support this concept. Pascal addresses this issue on compile time, but cannot handle it at run time.

https://doi.org/10.1371/journal.pone.0088941.g012

Another important consideration is a language's ability to detect errors related to pointer at compile time or run time. Certainly, for better diagnostics compile time detection of errors related to the pointers is much safer, and is very helpful for novice programmers as well. C/C++ have no support for detecting errors related to pointers at compile time, as well as at run time as shown in Figure 12 (line 2,3,4 -Code Listing 20). In Python all variables are considered as reference variables, which are always implicitly referenced, and direct access to the memory address is not allowed. Java does not have pointers, and only supports reference types, which can only point to objects. Java does not support pointer arithmetic on reference types. This in turn, reduces many error prone practices by novices. C# includes both references of Java and pointers of C++ using unsafe modifier possibly to provide backward compatibility with C and C++. Ada pointers are called access types and do not allow pointer arithmetic. Fortran77 does not have pointers, whereas, Modula-2 and Pascal also prone to pointer errors. Hoare [63] states about pointers, “their introduction into high-level languages has been a step backward from which we may never recover”.

Lastly, stack and heap overflows are also serious security concerns. Stack overflow mostly occurs when infinite/large recursive calls are made which consume whole memory stack. All the discussed languages suffer from this issue. Heap overflows occur when such a data array is created at runtime which requires more storage space than available memory in heap. None of the languages has been able to resolve this issues properly. All languages suffer from this issue as well. Java attempts to handle this issue by defining StackOverflowError in its exception hierarchy. Table 7 shows the evaluation of the considered languages based on their conformance to security.

https://doi.org/10.1371/journal.pone.0088941.t007

Feature Uniformity.

A language is considered to be feature uniform language if a proper subset of that language is not able to solve all problems that can be solved by whole set [6] [9] . Stroustrup [43] claims that in order to learn a programming language one has to learn a few fundamental constructs, techniques, and underlying models. The minimality of constructs certainly relaxes the learning curve. The feature uniformity in turn, can be anticipated in more than one ways, namely, feature exclusiveness and feature multiplicity [9] [21] , [62] .

Feature exclusiveness means a particular task can be accomplished by using exclusively different language constructs. As an example, swapping of two variables is a commonly taught programming problem in the initial programming courses. One can implement this problem in C++ using pointers, as well as using reference variables, as shown in Figure 13 (Code Listing 21). Method1 implements swap function with pointers and Method2 implements swap function through reference variables. In this case, if a novice programmer learns swapping through pointers, then in order to understand Method2, she has to learn the concept of reference variables as well, which in turn, increases the learning curve.

https://doi.org/10.1371/journal.pone.0088941.g013

Feature multiplicity means more than one ways to accomplish the same task while using the same language constructs [6] , [9] , [21] . As an example the Figure 14 (Code Listing 22) shows the feature multiplicity for incrementing the value of a variable; and assigning a value to an index of an array.

https://doi.org/10.1371/journal.pone.0088941.g014

The major problem caused by the unavailability of feature uniformity is that the size of the language increases, which in turn, results into a longer learning curve. Feature multiplicity can be observed in the control structures as they enhance language size by substituting each other. For example in C++ while and for loops are different in syntax but both have same semantics. Figure 1 shows the evolution of the programming languages, we can observe that Pascal was evolved from ALGOL60, hence, it carries the features of its predecessor language. This leads to an increase in the language complexity, in terms of the number and type of constructs. Resultantly, a language without feature uniformity usually demands the instructor to teach a subset of the language. A programmer can use this subset for writing code, but for reading other's code, comprehensive knowledge is required.

Therefore, a measure to identify whether a language holds the characteristic of feature uniformity is that its fully functional proper subset cannot be computed. Here, fully functional mean that a programmer can perform all tasks with the help of this subset. Table 8 shows the evaluation of feature uniformity for all leading FPLs.

https://doi.org/10.1371/journal.pone.0088941.t008

Less Effort for writing simple programs.

An FPL should require less programming effort to write simple programs. Furthermore, it should also support simple ways for input and output through console. In order to evaluate the effort to write simple programs we firstly consider the number of lines required to write a simple “Hello World!” program, and secondly, what is the required level of learning overhead [29] . The number of lines is fairly simple, however, we evaluate the second by identifying the number of distinct keywords used to write a simple program. For example, in Pascal writing a simple “Hello World!” program requires some learning overhead i.e. understanding of the keywords program , uses, begin, and end is required as shown in Figure 15 (Code Listing 23). In Ada, learning overhead ( use, with , procedure, begin, end) is required for “ Hello world! ” program as shown in Figure 15 (Code Listing 24). Modula-2 also requires learning overhead ( Module, import, from keyword, STextIO library ) for “Hello world!” program as shown in Figure 16 (Code Listing 25). Java programmers also have to learn many basic constructs of the language for writing simple program [12] , [30] . The main method used in Java and C# is complicated for a beginner, and is hard to explain to the novice programmers as it demands the explanation of the concepts like class, static, public, void etc. as shown in Figure 16 (Code Listing 26) and Figure 17(C ode Listing 27). Fortran requires learning the keywords program , end program and print as shown in Figure 17 (Code Listing 28). C/C++ have almost same learning overhead, as C++ requires the knowledge of namespace as shown in Figure 18 (Code Listing 29 and 30). Python learning curve is very simple for beginners as shown in Figure 19(C ode Listing 31).

https://doi.org/10.1371/journal.pone.0088941.g015

https://doi.org/10.1371/journal.pone.0088941.g016

https://doi.org/10.1371/journal.pone.0088941.g017

https://doi.org/10.1371/journal.pone.0088941.g018

https://doi.org/10.1371/journal.pone.0088941.g019

The other consideration for writing simple programs is the easier use of console I/O for primitive data types. The traditional pedagogical activities in teaching an FPL involve problems that include data input from the user, and display the output to the user through console. In order to evaluate the considered languages we have incorporated a simple I/O based computer program that inputs an integer ‘a’ from the user, and displays this integer with message in the format “value of a  = ” followed by the value of variable ‘a’ . Table 9 shows that the code for the aforementioned problem for all considered languages. We have shown multiple ways of input for Java and C#. We have evaluated it using two considerations, firstly the learning overhead that is based on the number of lexemes; and secondly, we rate the languages higher if they possess primitive constructs for I/O than the ones which use library functions for such purpose.

https://doi.org/10.1371/journal.pone.0088941.t009

The evaluation of console input as shown in Table 9 shows that Pascal, Modula-2, and C++ are the simplest for input. Fortran, Ada, and Python involve some additional constructs. C and C# involve even more constructs and concepts, lastly, Java offers most difficult way as it requires a lot of learning overhead for a novice.

Table 9 shows the output of most of the languages is much simpler as compared to the input methods. Pascal, Python, and C++ offer the simplest way to output data on console. Java, C#, and Fortran involve even more constructs for console output. C, Modula-2, and Ada require different instructions for different data types, which makes the output statement more complicated. Table 10 reflects the amount of effort needed to write simple programs in all considered FPLs. Table 10 shows the rating of the considered languages based on the number of language constructs to write a simple program, minimum number of constructs reflect low learning overhead.

https://doi.org/10.1371/journal.pone.0088941.t010

Environmental Features

In this section we discuss each environmental feature in detail. These environmental features have been evaluated by considering a language's conformance to their defining sub-features. Furthermore, these features have also been rated against the aforementioned four qualitative values.

Demand in Industry.

The industrial strength of a language is that it should genuinely be capable of being used for programming in realistic industrial and commercial situations. A number of FPLs are popular in the educational institutes because of their significance in the software industry [2] , [25] .

In order to evaluate the industrial relevance of a language we consider the following features: i) the number of code repositories available online for a particular language; (ii) the number of available jobs; and (iii) the number of web searches made for a language. Different data sources available on the Internet have been used to evaluate the aforementioned three parameters. The data from github.com ( https://github.com/munificent/github-language-ranking/blob/master/2013-08-01%20results.txt ) has been incorporated to get the statistics about the number of code repositories for a language. Secondly, we can find the job trends from jobstractor.com ( http://jobstractor.com/monthly-stats ) , which provides the statistics about the jobs advertised requiring expertise in a certain language. Lastly, we have incorporated the data from TIOBE index ( http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html ) that reflects the use of a language in web search. We believe that all these statistics help us in identifying the strength of a language for its usage and need in industry. Furthermore, all these indexes are kept up-to-date by their respective administrators, and hence can provide the language evaluator with the latest statistics, as well as, enable her to find the recent trends. We have also incorporated the latest statistics from all these data sources.

An important consideration is that all the indexes used in the evaluation of this parameter provide us quantitative data, therefore, we do not map this data on our proposed qualitative values, but we present the data in its actual form. However, we have given a special consideration to these values while computing the language's suitability score in the scoring function section. The recent statistics about code repositories, jobs, and number of web searches have been presented in Table 11 .

https://doi.org/10.1371/journal.pone.0088941.t011

It is important to note that by definition it is evaluated in quantitative terms, therefore, we do not map it to the above mentioned qualitative values. However, we treat it in a different manner, as explained in the scoring function section, while computing the overall score of a language.

Contemporary Features.

Contemporary programming features and methodologies are always appealing for both academia and industry; therefore the FPL should include contemporary features based on software engineering principles [10] . These features include support of: object oriented programming, multi-threading, exception handling, packages, generic programming [21] , [25] , [31] . Although all such features are not taught in the FPL course, yet these features are taught in the subsequent programming courses, and are widely required in industry. Certainly, choosing a language as FPL which possesses these features reduces the transition cost of learning yet another programming language in the advanced programming courses.

Object-oriented paradigm is closest to the real world applications and is easy to understand. Object orientation is a popular and demanded feature due to its conformance to nature, reusability, and easy implementation [18] , [27] , [34] . C, Modula-2 and Pascal are the only widely used FPLs which do not support object oriented paradigm, whereas, the current versions of all other languages support this paradigm. Most of the popular object oriented languages, particularly the ones considered in this research, are multi-paradigm languages [21] .

Concurrency in Programming language [64] , [65] occurs at instruction, statement, or subprogram level. Concurrency can be physical (i.e. more than one concurrent units runs simultaneously on multiple processors), or logical (i.e. more than one concurrent units run simultaneously on a single processors). In both cases, it requires synchronization, which is implemented in two ways: competition synchronization (mutually exclusive access to shared data), and cooperation synchronization (among competing tasks). All languages considered in this research work support concurrency to a certain extent. High performance Fortran specifies statements that can be executed concurrently, and also includes statements for distribution of data over memory units connected to multiple processors [66] . Ada, Java, and C# support monitors for mutually exclusive access, and semaphores for cooperation synchronization. Ada tasks are heavy weight tasks which communicate with each other using rendezvous mechanism. Java supports light weight concurrent units, any class that inherits Thread class or implements Runnable interface, and also override a method named run, can be executed concurrently as shown in Figure 20 (line 1,2 -Code Listing 33). Here, the competition synchronization is implemented through synchronized block or method, whereas, cooperation synchronization is implemented through notify , notifyAll , wait , join , sleep , and yield methods. C# further improves over Java as unlike Java it allows any method to be concurrent. Furthermore, it supports three types on thread synchronization namely, lock , interlock and monitor . Python concurrency model is loosely based on Java as illustrated in Figure 20 (Code Listing 32). Currently, Python thread class does not support priorities, thread groups, resume, suspend , and interrupt . C, C++, Modula-2, and Pascal partially support concurrency with the help of library functions.

https://doi.org/10.1371/journal.pone.0088941.g020

Exceptions are run time unusual events, erroneous or not, detectable by hardware or software and may require special processing [21] , [55] . Exception handling is a process done by code unit called exception handler. It increases the reliability as it avoids runtime failures that result into cascading aborts as mentioned in Figure 21 (line 6, 11, 15 - Code Listing 34). The write statement in procedure f2 will generate divide by zero exception for the instance where the variable p holds value 0. Delivering such programs to clients is highly unethical in software engineering practices. A good language should be equipped with proper exception handling mechanism.

https://doi.org/10.1371/journal.pone.0088941.g021

Among the considered FPLs Fortran, C, Pascal, and Modula2 do not support exception handling. Ada supports exception handling with many problems. First, Ada exception propagation model propagates exception to outer scope from where exception is not visible and it is hard to trace the origin of error propagation. Second, its exception handling for task is very weak; a task without exception handling dies or raises exception. Finally, it may not always be possible to determine the object which originated the exception [67] , [68] , [69] . C++ is the first C-based language that includes exception handling, where exceptions are not named and are connected to handlers using actual parameter type. Formal parameter may be omitted by using ellipsis (…), in which case it catches all types of exceptions. Primitive types can be used as formal parameters in handlers, but the best practice is to define user classes for exceptions in order to enhance readability. Java supports improved form of exception handling over C++ and Ada in many ways. Firstly, only those objects that are instance or descendent of Throwable can be thrown as an exception. Secondly, it improves readability by introducing checked exceptions using throws clause in method declaration. A method without throws cannot throw checked exception that it does not handle. Furthermore, introducing finally clause for cleanup actions also enhances program readability. Lastly, JVM implicitly catches and throws variety of exceptions that can be handled by other user programs. C# handles the exceptions identically as of Java except the fact that it does not support throws clause. Python supports exception handling using try, except, else, finally keywords.

Packages divide the program into manageable smaller pieces of code and provide easy ways to separately compile and assemble different pieces together to develop a large program without the inherent complexity due to its size. They also provide name encapsulation in order to define name scopes that assist in avoiding name conflicts in the APIs that expose a package to the user. Good modular design using packages supports minimum coupling and maximum cohesion [18] , [28] . In Java, package is a group related public types (classes, interfaces, enumerations, and annotation), where package name dictates the source file directory structure as presented in Figure 22 (Code Listing 35). One Java package can have many sub packages. Python and Ada support packages similar to that of Java. C++ and C# provide namespace as package which unlike Java does not depend on physical layout of files in directories on disk. as mentioned in Figure 22 (Code Listing 36). The rest of languages loosely define packages: e.g. C supports packages by inclusion of header files which are stored on disk; whereas, Pascal, Modula-2 and Fortran support packages in the form of modules.

https://doi.org/10.1371/journal.pone.0088941.g022

Generics are subprograms or abstract data types that take parameters of different types in different activations, and are also referred to as parameterized polymorphism [21] . The use of generics offers several advantages e.g. it prevents code duplication; helps early diagnostics of errors as it converts runtime errors to compile time errors; and programmers no longer have to manually cast elements [53] , [54] , [55] , [56] . In our considered languages Ada, C++, Java and C# support generics, while Python is dynamic strongly typed language which implicitly supports generics. Java implements generics ( Figure 23 , Code Listing 37) in the following different ways as compared to Ada and C++. Firstly, generic parameters must be classes not primitives. Secondly, only one copy of code is created without considering number of instantiations, called raw methods. Thirdly, restrictions can be applied to parameter that can be passed to generic method or abstract data type, called bounds. Finally, wildcards are also supported for any collection type.

https://doi.org/10.1371/journal.pone.0088941.g023

C# support generics in the similar way as of Java except there is no wild card support. Ada and C++ as shown in Figure 23 (Code Listing 38) support both generic subprograms and abstract data types. A separate copy of subprogram is created at compile time for each different type, while the binding of actual and formal parameters is static. Fortran has incorporated the support of generics in its recent versions. Modula-2, Pascal, and C do not support generics. Table 12 provides the details of our considered programming languages in terms of the discussed contemporary features.

https://doi.org/10.1371/journal.pone.0088941.t012

Easy Transition.

A good FPL should allow the transition to learn any new programming language in a smooth fashion. Concepts learned with the FPL should be easily transferable to another language [32] . As an example, if one learns C++ as FPL, then the transition to Java is very smooth, since both language share many similar constructs and furthermore, these languages have almost comparable syntax. In order to evaluate our considered languages for this feature, we use three parameters. Firstly, paradigm shift is considered, i.e. shifting from imperative to Object Oriented paradigm incurs one unit of cost, whereas, vice versa does not have any cost, as Object Oriented languages are imperative. Similarly, shifting from non-concurrent to concurrent language incurs one unit cost. Secondly, a transition bears one unit of cost if the source language is statically typed, and destination language is dynamically typed, and vice versa. Lastly, the evolution of languages presented in Figure 1 is considered, which shows the influences that one language has on the other languages.

As we are computing the transition cost and certainly the language with overall minimum cost should be ranked higher. In order to map these values to our proposed qualitative measures we define a simple criterion, which assigns a category to a language based on the overall transition cost of a language to all other languages, as shown in Table 13 , where ‘N’ is the number of the considered languages. The value of third parameter is equal to minimum number of hops (edges) between two languages in the evolution graph. For the total score per language we added up all values in the column see Table 14 , where we present all three costs in the following format: paradigm shift/static-dynamic type shift/hop count.

https://doi.org/10.1371/journal.pone.0088941.t013

https://doi.org/10.1371/journal.pone.0088941.t014

Readable Syntax.

The syntax of the language should be readable and consistent [26] , [33] . The regular cases of errors are discovered in programs only because the programmer does not understand code written by others due to its poor readability [3] . Both beginner and experienced programmers, take advantage of good readability. In particular, for the novice programmer, it makes the learning of the language easier, helps to reduce the number of errors, and makes the code easier to maintain [34] , [35] .

In order to evaluate the readability of a language we use the following three parameters: i) Identifier's name should neither be length dependent, nor declared implicitly; ii) Consistent compound statement; iii) Meaning of constructs is not context dependent.

The first evaluation parameter for readability ensures that the names of the identifiers should not be length dependent and there should not be any implicit declaration. e.g. in Fortran 77, the length of an identifier can have 6 characters at most. It also allows implicit declaration, and identifier names starting with I,J,K,L,M,N are implicitly declared to be integer, and others are considered as real. Python infers the type of all kind of variables based on the value assigned, whereas, in C# local variables can be given an inferred type of var instead of an explicit type.

There should be a clear and consistent syntax for each type of a compound statement. For instance, the usage of special keywords for signaling the start and end of each compound statement e.g. end if for if statement, end loop for loop termination, end procedure-name for procedures, and end program for program helps a great deal towards the better readability of a program. In our considered languages Fortran, Modula-2, and Ada support this feature. The code in Figure 24 (lines 4, 7, 11, 12 - Code Listing 39) shows how Ada supports this concept. The rest of the languages do not have corresponding ending keywords for compound statements, rather most of them use “{}” or “ begin end ” to represent a compound statement.

https://doi.org/10.1371/journal.pone.0088941.g024

Another consideration for the evaluation of readability is that the forms and meaning of construct should not be appearance or context dependent [21] . In our considered FPLs most of the languages adhere to this requirement, however, C, C++, and Fortran do not conform to this requirement. For example, in C static keyword has different meaning if declared inside and outside function. Similarly, in Fortran “INTEGER :: a” is considered as declaration statement for the declaration of variable ‘a’ of type Integer, whereas, “INTEGER  =  a” is an assignment statement where the value of a is assigned to another variable named INTEGER. C# loosely conforms to this requirement as it supports context dependent keywords. Such ambiguous semantics create poor readability for novices. Unary operators are strongly discouraged due to poor readability [18] , [36] as shown in Figure 25 (line 3 -Code Listing 40). Example in Figure 25 (line 4 - Code Listing 41) gives different results on different compilers. Table 15 shows the evaluation of the considered programming languages for readability.

https://doi.org/10.1371/journal.pone.0088941.g025

https://doi.org/10.1371/journal.pone.0088941.t015

Quality Coding Standards.

The main objective of the coding standard is maintainability. Other important things that relate to the strength of the quality standards include simplicity, consistency, portability, extensibility, clarity, safety, and correctness [51] , [52] . Stroustrup [41] states that production of quality code should be elevated to a central role in software development. When in doubt, the programmer should endeavor for clarity rather than efficiency [37] . The style of writing directly impacts the readability and understandability of the end product [38] . By enforcing languages to implement the coding standards we can save cost of code review and minimize the human dependency and obviously minimize the possibility of bad coding practices and explicitly improve the readability of our source code. Indentation, comments, braces, naming conventions, and parentheses are most commonly considered as quality coding standard attributes [37] , [38] , [39] , [49] . Certainly, it is imperative to introduce the novice programmers about the coding standards from the beginning [51] .

There exist coding standards like QP/C++ ™, MISRA-C++ [46] , [47] which provide a guideline for writing quality code. In this work, we evaluate the quality coding standards from the perspective of an FPL, we consider the following parameters: i) support for comments; ii) avoidance of dangling else problem; iii) use of proper naming convention; iv) use of parentheses in expressions.

All programmers including novices are encouraged to comment the code so as to remind themselves of the tricky logic implemented in a particular part of the code. Comments are also used to provide documentation of the code. Table 16 shows differernt types of comments supported by programming languages. Generally, different types of comments are useful, but the coding standards discourage block comments as they are prone to errors. One serious problem in block comment is that novice programmer may accidently forget to end the comment which may swallow useful code. It will be swallowed as shown in Figure 26 (code listing 42) (line 1-4).

https://doi.org/10.1371/journal.pone.0088941.g026

https://doi.org/10.1371/journal.pone.0088941.t016

Apparently, the comments may look to be a minor issue in a language; however, an unsafe comment format in a language may become a source of nasty errors particularly for novices. C++ uses/* characters for starting a block comment, while these characters are also used as multiply (*) operator, divide (/) operator, and pointer redirection. An unusual code may result into undesired erroneous programs as shown in Figure 27 (Code Listing 43), where comment syntax clashes with that of a pointer. The syntax/* denotes the start of comment and compiler will return error. Therefore, in this case the use of a space character is very significant. Correct code can be written with one space between/and * or use parentheses for *ptr as shown in Figure 27 (Code Listing 44). Therefore, the coding standards also discourage the usage of block comments [46] .

https://doi.org/10.1371/journal.pone.0088941.g027

Among different types of comments the end of Line Comment (In-Line Comments) is the most unambiguous and preferable comment [46] . Similarly, C offers mega comments, another type of comment, which helps in activation and deactivation of a particular part of code during execution. This type of comments is used for debugging the code. Furthermore, newer languages are equipped with documentation comments, which are useful in managing the documentation for language APIs.

In our evaluation we rate the languages higher if they facilitate the programmers with end-of-line, documentation, and mega comments, while we denounce the usage of block comments in a language. Although mega comments are useful, yet this is not considered among the mainstream types of comments. Hence, in our evaluation, we encourage the presence of mega comments in a language, but treat their absence unnoticed.

Based on the above discussion and supported code listings we conclude that the languages C, Modula-2, and Pascal partially support the comments. The reason is that C supports mega comment, but also has notorious block comment. Modula-2 and Pascal just support the block comment. Only Python supports clean comments and thus fully supports comments, while the rest of the languages have cleaner comments as well as block comments, and hence they adhere most of the comments.

It is highly recommended in major quality coding standards to use compound statement in if , else structures and, while and for loops. For example use compound statement after then and else part as shown in Figure 28 (Code Listing 45), even if there is only a single statement in else and then part, preferable style is to use compound statement as the coding standards. MISRA/C++ [46] and QP/C++ ™ [47] also define these rules, and such rules can be verified by checkers like PC-Lint [48] , and also protect code from dangling else problem as shown in Figure 28(C ode Listing 45).Most of the leading FPLs suffer from dangling else problem as shown in Figure 29 (line 9 - Code Listing 46), where an else statement links with unwanted if statement. Python solves dangling- else problem by requiring indentation of else keyword with its matching if keyword as shown in Figure 30(C ode Listing 47). Modula-2, Ada and Fortran solve the dangling else problem by using end if keywords. C, C++, C#, Pascal and Java suffer from dangling else problem.

https://doi.org/10.1371/journal.pone.0088941.g028

https://doi.org/10.1371/journal.pone.0088941.g029

https://doi.org/10.1371/journal.pone.0088941.g030

The use of proper naming conventions enhances readability and comprehensibility, and it reduces the maintenance of the code [39] . Approximately, 70% of the source code of a software system consists of identifiers [70] . Knuth noted that descriptive identifiers strongly indicate the code quality and comprehensibility [71] . Java quality standard follows different naming convention for class identifiers, variables, function names and constants as shown in Figure 31 (Code Listing 48). Unfortunately, naming conventions cannot be enforced by programming languages [70] , however some languages like Java and C# implicitly encourage the programmer to get used to quality naming conventions [50] , as both languages involve rigorous usage of APIs which follow the coding standards.

https://doi.org/10.1371/journal.pone.0088941.g031

Use of parentheses in expressions enhances readability and code quality. Most of the coding standards enforce writing parenthesis in expressions [49] , [52] , e.g. if the programmer wants to write c = (a+d)b it will generate error. Changing it with c = a+d*b requires proper understanding of operator precedence and associatively rules. Preferable style is to use proper parentheses in expression in order to enhance code quality as shown in Figure 32 (Code Listing 49). Parentheses can easily be enforced in programming language design. It will enhance readability and help saving the novice programmer from accidental use of operators without a strong knowledge of operator associativity and precedence.

https://doi.org/10.1371/journal.pone.0088941.g032

Some considerations in the coding standards cannot be enforced implicitly by a programming language but others can be enforced by changing language design as described by Table 17 .

https://doi.org/10.1371/journal.pone.0088941.t017

None of the leading FPLs completely supports quality coding standards that can be implemented with slight modification in syntax and semantics and leaves this issue to the software engineers. Due to poor quality of code a software engineer spends many hours on code inspection, debugging, and maintenance [10] . Table 18 shows the conformance of the considered languages to the quality coding standards.

https://doi.org/10.1371/journal.pone.0088941.t018

User-Friendly Integrated Development Environment.

In order to evaluate the user friendly integrated development environment (IDE) we consider the following parameters: i) structured editor; ii) pretty printer; iii) static checker; iv) debugger; v) novice programming environment. Every programming language must have a good graphically integrated development environment [7] . These environments facilitate both the novice and seasoned programmer to write, indent, and visualize the code easily. Furthermore, some researchers consider that for choosing an appropriate FPL, the programming environment alone would probably be as important as the programming language itself [7] , [18] , [40] .

Structured editing feature in source code editor helps the programmer avoid syntactic mistakes by automatically correction and by suggesting corrections. This helps novice programmer to concentrate on problem solving rather than focusing on syntactic issues [72] , [73] . All considered FPLs support this feature.

Pretty Printer handles the formatting of source code, also known as code beautifier, and involves indentation, lexeme coloring, font size adjustment, block collapse and expansion. The inclusion of such features in the text editor helps the novice programmer a great deal to improve the readability of a program [74] , [75] . Except Fortran, all languages are supported by the support of pretty printers.

Static checking involves the identification of unused variables, unused functions, and violation of custom naming conventions. This helps increasing the conformance of code to the coding standards. There are several checker tools for Java (Checkstyle, FindBugs, GrammaTech Code Sonar), C/C++(CppCheck, cpplint, lint, PC-Lint), Pascal (Undertstand), Ada (AdaControl, LDRA Testbed) and Python(Pychecker, Pylint). Except Fortran all other considered FPLs facilitate static checking.

The debuggers allow a programmer to examine the state of the variables at a certain point by stopping the execution of a program. The most widely offered features of a debugger are to add watch, insert breakpoints, running program step by step, and continue execution at different locations in program. This helps the novice programmers to find errors in their programs. Furthermore, it is highly recommended that the novice programmers should learn debugging strategies [76] , [77] . All considered FPLs are equipped with debuggers.

A novice visual programming environment is also imperative in terms of teaching and learning an FPL. There are several rich and user friendly novice programming environments for many considered FPLs. The list of some of the programming environments is as follows [83] : Ada (Lego Mindstorm, Ada GIDE), C/C++(BlockC, Ch), Fortan(GNOME), Java (BlueJ, CourseMaster, Greenfoot, Jeliot), and Pascal (Genie, GPCeditor, Emile, ModelIt), and Python(Alice98, Python Turtle). Some of the IDEs also support drag and drop coding (BlockC) options which help the novices to write code without syntax errors. Table 19 shows the ratings of our considered programming languages based on the provision of user friendly IDEs.

https://doi.org/10.1371/journal.pone.0088941.t019

Scoring Function

https://doi.org/10.1371/journal.pone.0088941.t020

Where, ‘n’ is the total number of parameters in the language evaluation framework, which in our defined framework are 13.

https://doi.org/10.1371/journal.pone.0088941.t021

Score Computation for the Considered Languages and Discussion

In this section, we compute the quantitative scores of the considered FPLs using the above mentioned scoring function. We obtain the scores against the parameters of each category based on the discussion in previous section. Table 22 shows the scores of all technical features for these languages; Table 23 shows the same for environmental features. Lastly, the scores of these features are combined while using the default weights in Table 24 . This table, in turn, shows the suitability score for each language.

https://doi.org/10.1371/journal.pone.0088941.t022

https://doi.org/10.1371/journal.pone.0088941.t023

https://doi.org/10.1371/journal.pone.0088941.t024

It is clear from Table 24 that Java has obtained overall highest score and thus, with default settings, it is the most suitable programming language using our defined scoring function. Python and Ada are next most suitable languages based on their obtained scores. However, one significant point is that Python is technically most equipped language as shown in Table 22 . The reason is that it shows its strengths in many technical features i.e. it is Orthogonal , High Level, and Secure language, and also requires Less Effort in Writing the code . Whereas, Java ranks highest, w.r.t. the environmental features as shown in Tables 23 , by a significant margin. This is because of the facts that Java is highly demanded in industry, supports most of the contemporary features, provides easy transitions to the other languages, and has sufficient support in the form of user friendly development environments.

Table 1 shows that C++ is still following Java as the most widely used FPL, whereas Table 24 shows that it only has 48% conformance to the defined framework with default settings. However, Table 23 shows that it has strong support from the perspective of environmental features, and Table 22 shows that it lacks strength from the technical feature set, as it relies on efficiency rather than reliability. However, the reason for its popularity lies in strong support for environmental factors, as it supports contemporary features, easy transition, and there are several use friendly IDEs for this language.

As the default weight settings do not conform to the original popularity index of the languages, so there should be a different weighting criterion. However, it is very hard to come up with a generic and correct weighting criterion. Therefore, the scoring function should be customizable and the user should be able to tune the weight of each feature based on her preferences. As an example, consider the fact that Ada holds 3 rd position in overall scoring, but is not being considered among highly used FPLs as of now, as shown in Table 1 . The most probable reason seems to be that it fails to create any impact from the perspective of Industrial Demands, as shown in Table 23 . Based on this observation a user may consider “demand in industry” and “easy transition” more important than the rest of the parameters, and assigns them weights of 3, and 2, respectively. Then, as shown in Table 25 , the ranks of C#, C++, and C are elevated, whereas, Ada, Modula-2, Pascal, and Fortran are degraded with this weighting scheme, while Java and Python are not affected on the ratings list, though their degrees of conformance is affected with the new weights. This certainly shows the strength of our proposed framework and scoring function, as it re-ranks the languages based on the customized settings. Hence, every user can look for an appropriate language based on her personal preferences. However, based on the discussion in the previous section, it is clear that the user of this framework should have a reasonable understanding of the language theory to evaluate the language from technical perspective, and should have up-to-date information about tools, and statistics related to the language to evaluate environmental factors. But, the anticipated users of this framework are the personnel who are either course instructors, or curriculum designers, who in our opinion, possess sufficient background knowledge to use and customize such frameworks.

https://doi.org/10.1371/journal.pone.0088941.t025

Finally, the defined framework and scoring function are generic and flexible in two ways: firstly, any language can be evaluated on the given criterion; secondly, we have equipped the framework with many features, however, more features can seamlessly be incorporated with new advancements in the domain of programming languages.

Conclusion and Future Directions

In this article, we have proposed a comprehensive framework for the evaluation of a programming language in terms of its suitability as an FPL. This framework mainly comprises of two main categories of features, namely technical and environmental feature sets. The technical features deal with the intrinsic and language theoretical aspects, whereas, the environmental features discuss the external factors. We have defined each feature and its corresponding sub-features, and have presented the evaluation criterion for each sub-feature. Furthermore, we have presented a scoring function to compute a quantified suitability score for a language based on our framework. In addition to this, based on the language evaluation and scoring function, we have computed the suitability scores for widely used FPLs. We have also used different parameter settings for computing the suitability scores, and give the users leverage to customize the weightage of different features, which reflects the flexibility of our proposed scoring function. This framework not only helps in the evaluation of a language as an FPL, but can also be used as a guideline for designing new FPLs.

The potential future directions of this work include the customization of a programming language so as to make it more suitable FPL w.r.t. the technical feature set. Similarly, our work highlights the drawbacks of the languages in terms of environmental feature set, and hence, new tools can be developed to increase the conformance of a language to the environmental features like quality coding standards, user friendly environment, and readability .

Supporting Information

Appendix s1..

https://doi.org/10.1371/journal.pone.0088941.s001

Acknowledgments

We are thankful to the anonymous reviewers whose valuable and constructive review comments helped us improving this manuscript.

Author Contributions

Conceived and designed the experiments: MSF SAK AA. Performed the experiments: MSF SAK AA. Analyzed the data: MSF SI FA AA. Contributed reagents/materials/analysis tools: MSF. Wrote the paper: MSF. Revised the manuscript: MSF SAK AA.

• 1. Stroustrup B (2009) Programming in an undergraduate CS curriculum, In Proceedings of the 14th Western Canadian Conference on Computing Education (WCCCE '09). ACM, New York: 82–89.
• View Article
• Google Scholar
• 3. Hoare CAR (1980) The Emperor's Old Clothes. ACM Turing Award Lecture. ACM.
• 4. Reid R J First-Course Language for Computer Science Majors, http://www.panix.com/~dictum/reference/language-lists/ . Accessed 5 December 2013.
• 7. McIver L (2002) Evaluating languages and environments for novice programmers. In 14th Workshop of the Psychology of Programming Interest Group:100–110.
• 8. Clarke S (2001) Evaluating a new programming language. In 13th Workshop of the Psychology of Programming Interest Group:275–289.
• 13. Gries D (1974) What should we teach in an introductory programming course?. In Proceedings of the 4th SIGCSE Technical Symposium on Computer Science Educatio. ACM Press :81–89.
• 14. Schneider GM (1978) The introductory programming course in computer science: ten principles. In Papers of the 9th SIGCSE/CSA Technical Symposium on Computer Science Education. ACM Press :107–114.
• 16. Roberts ES (1993) Using C in CS1: evaluating the Stanford experience. In Proceedings of the 24th SIGCSE Technical Symposium on Computer Science Education. ACM Press:117–121.
• 18. Kölling M, Koch B, Rosenberg J (1995) Requirements For A First Year Object-Oriented Teaching Language. Proceedings SIGCSE'95. Nashville. Tennessee. ACM Press:173–177.
• 20. Jones C (1996) Applied Software Measurement: Assuring Productivity and Quality. Second Edition. McGraw-Hill.
• 21. Sebesta R (2012) Concepts of Programming Languages. 10th Edition. Addison-Wesley.
• 22. Wirth N (2002) Pascal and Its Successors. Software Pioneers. Springer-Verlag.
• 23. Stroustrup B (1995) The Design and Evaluation of C++. Addison-Wesley.
• 24. Jacquot JP (2002) Which use for Java in introductory courses? In Proceedings of the second workshop on Intermediate representation engineering for virtual machines:119–124.
• 26. Wirth N (1975) An Assessment of the Programming language Pascal. IEEE Transaction on Software Engineering: 192–198.
• 27. Salus HP (1998) Handbook of Programming Languages Vol. I: Object Oriented Programming Languages. Macmillan Technical Publishing.
• 28. Eckel B (2000) Thinking in Java (Second Edition). Prentice Hall.
• 31. Pirkelbauer P, Dechev D, Stroustrup B (2010) Support for the Evolution of C++ Generic Functions. 3rd International Conference on Software language Engineering. 123–142.
• 33. Wirth N (1974) On Design of the programming language, IFIP CONGRESS. 386–393.
• 35. Dijkstra E (1979) The humble programmer. In Classics in software engineering. Yourdon Press:111–125.
• 37. The Corelinux Consortium (2000) C++ Coding Standards http://www.literateprogramming.com/cppstnd.pdf . Accessed 9 April 2013.
• 38. Johnson K A, Pettit IV, Opdahl S B (1997) Ada 95, Quality and Style: Guidelines for Professional Programmers. Lectures Notes in Computer Science. Springer.
• 39. Java Code Conventions by Sun Microsystems (1997) www.oracle.com/technetwork/java/codeconventions-150003.pdf . Accessed 5 April 2013.
• 43. Stroustrup B (2012) Foundations of C++. Proc. 22nd European Symposium on Programming (ESOP). Springer LNCS:1–25.
• 46. MISRA (2008) C++ Guidelines for the use of the C++ language in critical systems. ISBN 978-906400-03-3.
• 47. Samak M (2008) Practical UML Statecharts in C/C++. Event-Driven Programming for Embedded Systems. Second Edition. Newnes.
• 48. PC-lint for C/C++ (1985) http://www.gimpel.com/html/index.htm . Accessed 7 September 2013.
• 49. C# Coding Conventions (C# Programming Guide) http://msdn.microsoft.com/en-us/library/vstudio/ff926074.aspx . Accessed 10 November 2013.
• 50. Philips Healthcare (2009) C# Coding Standard. Version 2.0.Philips Healthcare.
• 51. Li X, Prasad C (2005) Effectively teaching coding standards in programming. In Proceedings of the 6th conference on Information technology education:239–244.
• 52. General Software Development Standards and Guidelines (2007) Version 3.5. NOAA National Weather Service NWS/OHD. Science Infusion Software Engineering Process Group (SISEPG).
• 53. Donovan A, Kiezun A, Tschantz MS, Ernst MD (2004) Converting java programs to use generic libraries. In OOPSLA '04: Proceedings of the 19th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications.
• 54. Vandevoorde D, Josuttis N (2002) C++ templates: the Complete Guide. Addison-Wesley Professional.
• 55. Stroustrup B (2013) The C++ Programming Language. 4th edition. Addison-Wesley Professional.
• 56. Parnin C, Bird C, Murphy-Hill E (2011) Java generics adoption: how new features are introduced, championed, or ignored. In Proceedings of the 8th Working Conference on Mining Software Repositories ACM:3–12.
• 57. Ryder BG, Soffa ML, Burnet M (2005) The impact of software engineering research on modern progamming languages. ACM Trans. Softw. Eng. Methodol. 431–477.
• 58. Denvir BT(1979) On orthogonality in programming languages. SIGPLAN Not:18–30.
• 59. The Ada Compiler Validation Capability (ACVC) Test Suites: http://archive.adaic.com/docs/flyers/acvc.html . Accessed 2 december 2013.
• 60. Boxing and Unboxing (C# Programming Guide) (2013) http://msdn.microsoft.com/en-us/library/vstudio/yz2be5wk.aspx . Accessed 8 december 2013.
• 61. Autoboxing and Unboxing (The Java™ Tutorials). http://docs.oracle.com/javase/tutorial/java/data/autoboxing.html . Accessed 6 december 2013.
• 62. Tucker AB, Noonan R (2009) Programming Languages. 2nd edition. McGraw-Hill.
• 63. Hoare CAR (1973) Hints on Programming Language Design. Technical Report. Stanford University. Stanford. CA. USA.
• 64. Ari B, Mordechai B (1982) Principles of Concurrent Programming. Prentice-Hall.
• 68. Weimer W, Necula GC (2008) Exceptional situations and program reliability. ACM Trans. Program. Lang. Syst. 30(2): .51 pages.
• 71. Knuth D (2003) Selected papers on computer languages. In: Center for the Study of Language and Information (CSLI Lecture Notes. no. 139). Stanford. CA.
• 72. Vollebregt T, Kats LC, Visser E (2012) Declarative specification of template-based textual editors. In Proceedings of the Twelfth Workshop on Language Descriptions, Tools, and Applications. ACM. USA. Article 8. 7 pages.
• 74. Rendel T, Ostermann K (2010) Invertible syntax descriptions:unifying parsing and pretty printing. In Proceedings of the third ACM Haskell symposium on Haskell. ACM:1–12.
• 75. Reiss SP (2007) Automatic code stylizing. In Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering. ACM. USA:74–83.
• 76. Bennedsen J (2008) Teaching and learning introductory programming—A model-based approach. PhD thesis. University of Oslo.
• 77. Bennedsen J, Schulte C (2010) BlueJ Visual Debugger for Learning the Execution of Object-Oriented Programs? Trans. Comput. Educ. 10(2). Article 8. 22 pages.
• 79. Scoy FV (2006). The Reid List 25. http://groups.google.com/group/comp.edu/browse_thread/thread/4f00b5f437ce261a/3267514419052033?q=Reid+List#3267514419052033 , Accessed 2 december 2013.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Correspondence
• Published: 19 July 2021

A Python-based programming language for high-performance computational genomics

• Ariya Shajii 1   na1 ,
• Ibrahim Numanagić 1 , 2   na1 ,
• Alexander T. Leighton 1 , 3 ,
• Haley Greenyer 2 ,
• Saman Amarasinghe   ORCID: orcid.org/0000-0002-7231-7643 1 &
• Bonnie Berger   ORCID: orcid.org/0000-0002-2724-7228 1 , 3  

Nature Biotechnology volume  39 ,  pages 1062–1064 ( 2021 ) Cite this article

11k Accesses

10 Citations

216 Altmetric

Metrics details

• Data processing
• Programming language

To the Editor — The vast growth of next-generation sequencing data has provided us with a new understanding of many biological phenomena. As sequencing technologies evolve, sequencing datatypes (such as standard Illumina short reads, PacBio long reads or 10x Genomics barcoded reads) typically require new implementations of corresponding computational analysis techniques, necessitating software that is not only computationally efficient, but also quick to develop and easy to maintain so as to enable rapid adaptations to new kinds of data.

However, developing an efficient software tool requires domain expertise in performance engineering, computational modeling, and the ability to translate biological assumptions into algorithm and software optimizations 1 . As a result, most high-performance genomics software is highly application specific, difficult to understand, and difficult to maintain. These factors collectively have sparked a reproducibility crisis in the fields of computational biology and life sciences in general 2 , 3 , 4 . Ultimately, researchers in the field lack good tools for developing and updating software; one typically must choose between a software ecosystem that allows rapid development at the expense of performance and scalability (for example, Python or R) or low-level languages that have higher performance but are harder to develop and maintain (for example, C, C++ or Rust) 5 . Existing solutions that try to fill the void between these extremes (for example, Matlab or Julia) are primarily geared toward numerical computing rather than computational genomics, and the few attempts that have been made at simplifying the development of bioinformatics software (for example, Rust-Bio 6 , SeqAn 7 , 8 , BioJulia 9 or Biopython 10 ) are based on languages that are either unfamiliar 11 , complex and hard to program, or too high level to attain good performance.

This is a preview of subscription content, access via your institution

Relevant articles

Open Access articles citing this article.

Sequre: a high-performance framework for secure multiparty computation enables biomedical data sharing

• Haris Smajlović
• , Ariya Shajii
•  …  Ibrahim Numanagić

Genome Biology Open Access 11 January 2023

Fast characterization of segmental duplication structure in multiple genome assemblies

• Hamza Išerić
• , Can Alkan

Algorithms for Molecular Biology Open Access 18 March 2022

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Yu, Y. W., Daniels, N. M., Danko, D. C. & Berger, B. Cell Syst. 1 , 130–140 (2015).

Article   CAS   Google Scholar  

Peng, R. D. Science 334 , 1226–1227 (2011).

Baker, M. Nature 533 , 452–454 (2016).

Lee, R. S. & Hanage, W. P. Lancet Microbe https://doi.org/10.1016/S2666-5247(20)30028-8 (2020).

Perkel, J. M. Nature 588 , 185–186 (2020).

Köster, J. Bioinformatics 32 , 444–446 (2016).

Article   Google Scholar  

Döring, A., Weese, D., Rausch, T. & Reinert, K. BMC Bioinformatics 9 , 11 (2008).

Reinert, K. et al. J. Biotechnol. 261 , 157–168 (2017).

Ward, B. J. BioJulia https://biojulia.net (accessed 19 November 2020).

Cock, P. J. et al. Bioinformatics 25 , 1422–1423 (2009).

Russell, P. H., Johnson, R. L., Ananthan, S., Harnke, B. & Carlson, N. E. PLoS One 13 , e0205898 (2018).

Stajich, J. E. et al. Genome Res. 12 , 1611–1618 (2002).

Li, H. Preprint at https://arxiv.org/abs/1303.3997 (2013).

Yorukoglu, D., Yu, Y. W., Peng, J. & Berger, B. Nat. Biotechnol. 34 , 374–376 (2016).

Hach, F. et al. Nucleic Acids Res. 42 , W494–W500 (2014).

Li, H. Bioinformatics 34 , 3094–3100 (2018).

Smith, T., Heger, A. & Sudbery, I. Genome Res. 27 , 491–499 (2017).

McKenna, A. et al. Genome Res. 20 , 1297–1303 (2010).

Bray, N., Dubchak, I. & Pachter, L. Genome Res. 13 , 97–102 (2003).

Berger, E. et al. Nat. Commun. 11 , 4662 (2020).

Berger, E., Yorukoglu, D. & Berger, B. International Conference on Research in Computational Molecular Biology 28–29 (Springer, 2015).

Abelson, H. & Sussman, G. J. Structure and Interpretation of Computer Programs (MIT Press, 1996).

Shajii, A., Numanagić, I., Baghdadi, R., Berger, B. & Amarasinghe, S. Proc. ACM Program. Lang. 3 , 125:1–125:29 (2019).

Download references

Author information

These authors contributed equally: Ariya Shajii, Ibrahim Numanagić.

Authors and Affiliations

Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA

Ariya Shajii, Ibrahim Numanagić, Alexander T. Leighton, Saman Amarasinghe & Bonnie Berger

Department of Computer Science, University of Victoria, Victoria, British Columbia, Canada

Ibrahim Numanagić & Haley Greenyer

Department of Mathematics, Massachusetts Institute of Technology, Cambridge, MA, USA

Alexander T. Leighton & Bonnie Berger

You can also search for this author in PubMed   Google Scholar

Corresponding authors

Correspondence to Saman Amarasinghe or Bonnie Berger .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Peer review information Nature Biotechnology thanks Ivan Costa and Judith Zaugg for their contribution to the peer review of this work.

Supplementary information

Supplementary information.

Supplementary Notes 1–4, including Figs. 1–5 and Tables 1–8

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Shajii, A., Numanagić, I., Leighton, A.T. et al. A Python-based programming language for high-performance computational genomics. Nat Biotechnol 39 , 1062–1064 (2021). https://doi.org/10.1038/s41587-021-00985-6

Download citation

Published : 19 July 2021

Issue Date : September 2021

DOI : https://doi.org/10.1038/s41587-021-00985-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

• Ariya Shajii
• Ibrahim Numanagić

Genome Biology (2023)

Navigating bottlenecks and trade-offs in genomic data analysis

• Bonnie Berger
• Yun William Yu

Nature Reviews Genetics (2023)

Algorithms for Molecular Biology (2022)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.