Lecture 2: Python Tokens, Variables, and Literals

Python Tokens and Character Sets

In Python, every program is formed using valid characters and tokens. The character set defines which characters are allowed in a Python program, while tokens represent the smallest meaningful units such as keywords, identifiers, literals, operators, and symbols.

Character Set

A character set is the collection of valid characters that a programming language understands. Python supports a wide range of characters, making it flexible and easy to use. Python character set includes:

• Alphabets: A–Z, a–z
• Digits: 0–9
• Special symbols: + - * / % = @ # $ & _ etc.
• Whitespace characters: space, tab, newline
• Unicode characters: Python supports full Unicode

These characters are used to form keywords, variables, expressions and statements.

Tokens

A token is the smallest meaningful unit in a Python program. Python code is interpreted by breaking it into tokens. Python has the following types of tokens:

1. Keywords

Keywords are reserved words with special meaning in Python. They cannot be used as variable or function names. Examples of keywords: if, else, for, while, break, continue, True, False, import, class

for x in range(1, 6):
   if x < 4:
       continue
   break

Here, for, if, continue and break are keywords.

2. Identifiers

Identifiers are names given to variables, functions, classes, etc. Rules for identifiers:

• Can contain letters, digits, and _
• Cannot start with a digit
• Cannot be a keyword
• Case-sensitive

name = "Python"
_count = 10
# Valid identifiers: name, _count
# Invalid identifiers: 2num, my-name, for

3. Literals (Values)

Literals are the fixed values or data items used in a source code. Python supports different types of literals:

3.1 String Literals

Represent text values enclosed in quotes.

msg = "Hello Python"
print(msg)

Output: Hello Python

3.2 Numeric Literals

Represent integer or decimal numbers.

a = 10
b = 3.5
print(a)
print(b)

Output:
10
3.5

3.3 Boolean Literals

Represent logical values: True or False.

is_valid = True

3.4 Special Literal

None represents the absence of a value.

value = None

3.5 Collection Literals

Represent grouped data such as lists, tuples, dictionaries and sets.

a = [1, 2, 3]  # List
tup = (1, 2)   # Tuple
d = {"a": 1}  # Dictionary
s = {1, 2, 3}  # Set

print(a)
print(tup)
print(d)
print(s)

Output:
[1, 2, 3]
(1, 2)
{'a': 1}
{1, 2, 3}

4. Operators

These are the tokens responsible to perform an operation in an expression. The variables on which operation is applied are called operands.

a = 5
b = 2
print(a + b)  # Addition
print(~a)     # Bitwise NOT

Output:
7
-6

5. Punctuators

These are the symbols that are used in Python to organize the structures, statements, and expressions. Some of the Punctuators are: [ ] { } ( ) @ -= += *= //= **== = , etc.

Python Variables and Naming

In Python, variables are used to store data that can be referenced and manipulated during program execution. A variable is essentially a name that is assigned to a value.

• Unlike Java and many other languages, Python variables do not require explicit declaration of type.
• The type of the variable is inferred based on the value assigned.

x = 5
name = "Samantha" 
print(x)
print(name)

Output:
5
Samantha

Rules for Naming Variables

To use variables effectively, we must follow Python's naming rules:

• Variable names can only contain letters, digits and underscores (_).
• A variable name cannot start with a digit.
• Variable names are case-sensitive like myVar and myvar are different.
• Avoid using Python keywords like if, else, for as variable names.

Below listed variable names are valid:

age = 21
_colour = "lilac"
total_score = 90

Below listed variables names are invalid:

# 1name = "Error"  # Starts with a digit
# class = 10     # 'class' is a reserved keyword
# user-name = "Doe"  # Contains a hyphen

Assigning Values to Variables

Basic Assignment: Variables in Python are assigned values using the = operator.

x = 5
y = 3.14
z = "Hi"

Dynamic Typing: Python variables are dynamically typed, meaning the same variable can hold different types of values during execution.

x = 10
x = "Now a string"

Multiple Assignments

Assigning Same Value: Python allows assigning the same value to multiple variables in a single line, which can be useful for initializing variables with the same value.

a = b = c = 100
print(a, b, c)

Output: 100 100 100

Assigning Different Values: We can assign different values to multiple variables simultaneously, making the code concise and easier to read.

x, y, z = 1, 2.5, "Python"
print(x, y, z)

Output: 1 2.5 Python

Type Casting a Variable

Type casting refers to the process of converting the value of one data type into another. Python provides several built-in functions to facilitate casting, including int(), float() and str() among others. Basic casting functions are:

• int(): Converts compatible values to an integer.
• float(): Transforms values into floating-point numbers.
• str(): Converts any data type into a string.

s = "10" 
n = int(s)

cnt = 5
f = float(cnt)

age = 25
s2 = str(age) 

print(n) 
print(f) 
print(s2)

Output:
10
5.0
25

Type of Variable

In Python, we can determine the type of a variable using the type() function. This built-in function returns the type of the object passed to it.

n = 42
f = 3.14
s = "Hello, World!"
li = [1, 2, 3]
d = {'key': 'value'}
bool = True

print(type(n))  # 
print(type(f))  # 
print(type(s))  # 
print(type(li)) # 
print(type(d))  # 
print(type(bool)) # 

Concept of Object Reference

Let us assign a variable x to value 5.

x = 5

When x = 5 is executed, Python creates an object to represent the value 5 and makes x reference this object.

Now, let's assign another variable y to the variable x.

y = x

This statement creates y and references the same object as x, not x itself. This is called a Shared Reference, where multiple variables reference the same object.

Now, if we write

x = 'Python'

Python creates a new object for the value "Python" and makes x reference this new object.

The variable y remains unchanged, still referencing the original object 5. Now, If we assign a new value to y:

y = "Computer"

Python creates yet another object for "Computer" and updates y to reference it.

The original object 5 no longer has any references and becomes eligible for garbage collection.

Deleting a Variable

We can remove a variable from the namespace using the del keyword. This deletes the variable and frees up the memory it was using.

x = 10
del x
print(x)  # This will raise a NameError

Explanation:

• del x removes the variable x from memory.
• After deletion, trying to access the variable x results in a NameError indicating that the variable no longer exists.

Practical Examples

1. Swapping Two Variables

Using multiple assignments, we can swap the values of two variables without needing a temporary variable.

a, b = 5, 10
a, b = b, a
print(a, b)

Output: 10 5

2. Counting Characters in a String

Assign the results of multiple operations on a string to variables in one line.

word = "Python"
length = len(word)
print("Length of the word:", length)

Output: Length of the word: 6