memo: Python

args & kwargs

(2022-08-05)

*args：表示接受任意个数的 Positional arguments，然后存放入一个元组中；传参时位置要对应，例如

1
2
3
4
5


def fun (*args) 
    print(args)

fun('fruit', 'animal', 'human')
>>> 'fruit'，'animal'，'human'

**kwargs：表示接受任意长的 Keywords arguments，然后存放入一个字典中；传参时要对应 key，例如

1
2
3
4
5


def fun(**kwargs):  
    for key, value in kwargs.items():
        print("%s:%s" % (key,value)

fun(a=1,b=2,c=3)会输出 a=1 b=2 c=3

Positional args and keywords args

1
2
3
4
5
6


def foo(a, b, **kwargs):
    var1 = a
    var2 = b
    args = kwargs

    var3 = args.['key1']

注解 annotation

bilibili

类型不匹配警告

1
2
3
4


from typing import *

def func(a: int, b: List[int]): # 第一个参数是int，第二个参数是list，其中每个参数是int
    ...

1
2
3
4
5
6


def func(a: List[List[int]],    # 二维int数组
         b: Dict,
         c: Set[int],
         d: Tuple[int]
         ): 
    ...

1
2
3
4
5
6
7
8
9


class TreeNode:
    def __init__(self):
        self.val = None
        self.left = None
        self.right = None

    def dfs(node: TreeNode):    # 类型为类名，或"TreeNode"
        node.val = 1
        ...

1
2
3
4
5


def func(x: float) -> str:  # 返回值类型
    return str(x)

def func2(func1: Callable[[float], str]):
    ...

1

a: int = 15     # 变量注解

1

print(func.__annotations__)     # 查看函数的注解类型

Class Variable & Instance Variable

1
2
3
4
5


class MyClass():
    class_var = 1   # Class Variable

    def __init__(self):
        self.var = 123  # Instance Variable

All instances of the class have access to class_var, as well as the class itself.
If class_var is mutable type, like list, class_var will influence on all instances.

Reference: Python Class Attributes: An Overly Thorough Guide

@classmethod

classmethod can only be called through the class name instead through an object instantiated from the class, because it gets passed the cls as the argument.
- Similarly, self means a method belongs to an instance, and can only be called via instance.
classmethod can access the class variables.

classmethod usually is used for instantiating an object with a set of specific arguments rather through __init__().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


class PositionalEncoding(torch.nn.Module):
  def __init__(self, ..): # instance method
    ...

  @classmethod
  def from_conf(cls, conf, d_in=3):   # conf: 'default.conf'.model.code{}
      # PyHocon construction
      return cls(
          conf.get_int("num_freqs", 6),
          d_in,
          conf.get_float("freq_factor", np.pi),
          conf.get_bool("include_input", True),
      )

Code snippest from pixel-nerf

Ref: An essential guide to Python class methods and when to use them

@staticmethod

@staticmethod in Python - AskPython

No need to pass an instance (self) or class (cls) to it as the first argument.
So it can be called both by the class name and an instance.
Can be overridden by children class.

name

当一个 py 文件被直接执行时，__name__ 被设置为 '__main__'，如果是被 impot，则它的 __name__ 被设置为它的文件名(不包括后缀)，并且被导入时会执行顶层的代码（不包括 function, class）

freeCodeCamp-Goran Aviani

init

References:

为什么要写 init.py ? | Python - Hucci写代码

Notes:

(2025-03-26)

To differenciate two scenarios: import <a file> and import <a package> ^r1-Hucci

file

(2024-02-28) Ref: GfG

The file path of the function’s definition

1
2
3


# Hello.py
def HelloWorld():
  print("This is Hello.py")

Call that module in another file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


# GFK.py
import Hello
import os

Hello.HelloWorld()

print(Hello.__file__)

print(f'GFK.py\'s __file__: {__file__}')

print(os.path.join(os.path.dirname(os.path.abspath(__file__)), "third_party/glm/"))

Output of python GFK.py

1
2
3
4


This is Hello.py
/home/yi/Downloads/ipynb_test/test_file_/Hello.py
GFK.py's __file__: GFK.py
/home/yi/Downloads/ipynb_test/test_file_/third_party/glm/

Built-in functions

Python Docs

dir(object) returns a list of the names of attributes and methods of any kind of object, e.g., module, class, instance. No values are stored.

vars(object) returns a dict: object.__dict__ containing changeable attributes.

vars() acts like locals(): a dict of local variables for read.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


class person:
    name = "jack"

dir(person)     # A list, len = 27
person.__dir__  # A method
vars(person)    # same as person.__dict__, len=5

# instance
student = person()
setattr(student, 'age', 14)
vars(student)   # out: {'age': 14}
dir(student)    # a list of attr and methods of the object
set(student.__dir__()) - set(dir(person)) # out: {'age'}
set(student.__dir__()) == set(dir(student))  # True

set(dir(student)) == set(person.__dir__(student))  # True

set(dir(student)) - set(person.__dir__(person)) # {'__weakref__', 'age', 'name'}

set(person.__dir__(person)) - set(dir(student)) # {'__abstractmethods__', '__base__', '__bases__', ...

[::-1] flip image

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


import matplotlib.pyplot as plt
from PIL import Image # pillow
import cv2

Image.open('1.jpg') # RGB

cv2.imread('1.jpg') # nd array, BGR
plt.imshow(cv2.imread('1.jpg'))

plt.imshow(cv2.imread('1.jpg')[:,:, ::-1]) # 最后的通道取反

plt.imshow(cv2.imread('1.jpg')[::-1,:, ::-1]) # 上下翻转

plt.imshow(cv2.imread('1.jpg')[::-1, ::-1, ::-1]) # 左右也翻转

表达式内赋值

PEP 572, Python 3.8+，类似 n := xxx

比如，有一个字符串数组，想去掉每个字符串里的空格和空字符串

1
2
3


arr = ['a', 'bb ', '   ']
arr = [x.strip() for x in arr] # 删除字符串前面和后面的空格
arr = [x for x in arr if x] # 空字符串会当作false

两行写一行: arr = [ y for x in arr if (y := x.strip)) ]

团队开发最好还是用易懂的写法。 1_00_00什么意思？x:=y？Python里面的几个有趣特性

Assignment expression - RealPython

Index value from dict

Get value by giving index from a dictionary

Accessing dictionary value by index in python

Python Dictionaries are Ordered from python 3.7 6, so the index can be used like:

value_at_index = list(dic.values())[index]

Python Dictionaries are Ordered now, but how?…and why?

Make one-hot matrix

Convert a vector to one-hot matrix

Convert array of indices to one-hot encoded array in NumPy - StackOvf

1
2
3


values = [1, 0, 3]
n_values = np.max(values) + 1
np.eye(n_values)[values]

if/else in a list

comprehension SO

Flatten a list of list

Python - Flatten a list of lists to a single list (Dead link now)

Concatenate list of tensor: torch.stack() first, then reshape

Redirect print content

(2023-08-08)

Redirect the standard output (print stream) to a file using the sys module.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


import sys

# Save the current stdout stream to a variable
original_stdout = sys.stdout

# Write to file
with open('output.txt', 'w') as f:
  sys.stdout = f
  print("The content given to sys.stdout \
         will be redirected to file, \
         instead of being printed out")

# Restore the original stdout stream
sys.stdout = original_stdout

argparse

(2023-08-28)

action="store_true", this option is True when this argument appears in the command
nargs=+, this argument at least be assigned with 1 value. Similarly, nargs=3, the argument must be assigned with 3 values.

Argparse: how to handle variable number of arguments (nargs=’*’) - SO

Example:
1

parser.add_argument('--img_wh', nargs="+", type=int, default=[1152, 864])
The command line: python eval.py --img_wh 640 512. The args in vscode “launch.json”: "args": ["--img_wh", "1152", "864",]

Update Dict attributes (packge: mmcv)

mmcv Docs

mmsegment配置参数说明（五）- CSDN

1
2
3
4
5
6
7


from mmcv import DictAction
parser = argparse.ArgumentParser(description='Train a recognizer')
parser.add_argument('--cfg-options', nargs='+', action=DictAction,
    default={},
    help='override some settings in the used config, the key-value pair '
    'in xxx=yyy format will be merged into config file. For example, '
    "'--cfg-options model.backbone.depth=18 model.backbone.with_cp=True'")

Example in launch.json of adapt-image-models:

1
2
3
4


"args": [
    "--cfg-options", "model.backbone.pretrained=openaiclip", 
                     "work_dir=work_dirs_vit/diving48/debug",
]

Set choices:

1
2
3
4
5


parser.add_argument(
    '--launcher',
    choices=['none', 'pytorch', 'slurm', 'mpi'],
    default='none',
    help='job launcher')

Accept a dict

(2023-08-31)

Accepting a dictionary as an argument with argparse and python (duplicate) -SO

argparse don’t have a type of dict, there’re 2 workarounds:

Set type=str, then use json.loads() to parse:

Set type=json.loads:

1
2
3
4
5


import json
parser.add_argument('-d', '--my-dict', type=json.loads)
args = parse.parse_args()

mydict = args.my_dict  # Will return a dictionary

The DictAction of mmcv will only parse the top-level as a dict (key-value), but if the value passed is a dict too, it’ll be parsed as a string.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


"args": [
"--cfg-options", "model.backbone.num_frames=3",

  // "train_pipeline[1].clip_len=3",  // Cannot identify [1]

  // "dict(train_pipeline={'1': dict(clip_len=3)})", // Error: has to 2 params, but got 1.

  // "train_pipeline= {\"1\": {\"clip_len\": \"3\"}}" 
  // The value will be parsed as a string, which will override the whole previous dict.
 ]

Access a dict

myDict.update({'aKey': "newVal"}) or myDict.update(aKey='newVal')
myDict.get('aKey', retVal)
myDict.setdefault('aKey', 'defVal'): if 'aKey' doesn’t exist, it will be inserted with the default value defVal.
myDict.copy() or newDict = dir(myDict) are shallow copy: when change the no-primitive data (such as list, nested data), the orignal dict will also change, because it only duplicate the top-level data, while the low-level data is referenced to the original data. Copy a Python Dictionary: A Complete Guide • datagy
myDict.pop('aKey') will return and remove an item.
Delete items from dictionary while iterating Have to iterate the dict twice, the 1st round collects the keys to be deleted, and the 2nd round del those items.

logging

Real Python

1
2
3
4
5
6
7


import logging

logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
logger=logging.getLogger(__name__)
logger.setLevel(logging.INFO)

logger.info("This is an info message")

Output: INFO:__main__:This is an info message

logging.info doesn’t show up on console

tqdm

(2023-10-20)

DDG with searching “avoid writting tqdm bar into the log file”

Use tqdm to iterate the outer loop and print in the inner loop. How can I make the tqdm progress bar be printed less frequently in the log file?
1 2 3

for outer in tqdm(range(0, 1e5, 1e4)): for inner in range(1e4): print(outer+inner)

Output tqdm to a null device, and print/log the status bar after desired steps. python - Making tqdm write to log files - Stack Overflow

1
2
3
4


tqdm_bar = tqdm(range(20), file=open(os.devnull, 'w'))
for i in tqdm_bar:
  if tqdm_bar.n % 5 ==0:
      print(str(tqdm_bar))

Redirecting console logging to tqdm.write(), leaving other logging handlers (e.g. log files) unaffected. tqdm.contrib.logging - Docs

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


import logging
from tqdm import trange
from tqdm.contrib.logging import logging_redirect_tqdm

LOG = logging.getLogger(__name__)

if __name__ == '__main__':
  logging.basicConfig(level=logging.INFO)
  with logging_redirect_tqdm():
    for i in trange(9):
      if i == 4:
        LOG.info("console logging redirected to `tqdm.write()`")
  # logging restored

Discussion: Pass progress bar to logger · Issue #313 · tqdm/tqdm

tqdm output to stderr, which is not in sync with stdout. So only redict the stdout (print) to file: python test.py > output.log

How to “flush” tqdm progress bar explicitly?

Found in Google
Monitor log file in real-time: tail -f output.log. reddit
Disable tqdm bar
- Presetting argument. Inspired by the comment of this answer of Why is tqdm printing to a newline instead of updating the same line?; Searched by DDG with searching “tqdm leave”
  1 2 3 4 5
  
  from functools import partial from tqdm import tqdm tqdm = partial(tqdm, disable=True) for i in tqdm(iterable):
- Disable tqdm bar by environment variable after version 4.66.0:
  1 2
  
  # pip install tqdm --upgrade eport TQDM_DISABLE=1
  Silence tqdm’s output while running tests or running the code via cron; Searched by DDG with searching “tqdm disable”
Env variable overrides supported by tqdm==4.66.0
leave=False means the bar will disappear instead of leaving it left after finishing. How to remove progressbar in tqdm once the iteration is complete

Use Progress bar as status bar showing current entry, by removing all field but the description field and specifying position.

1
2
3
4
5
6
7
8


outer = tqdm.tqdm(total=len(files), desc="Files", position=0)

cur_file = tqdm.tqdm(total=0, position=1, bar_format="{desc}")

for name in files:
  video = imageio.mimread(name, memtest=False)
  cur_file.set_description_str(f"Current file: {name}")
  outer.update(1)

Progress bar and status logging in python with tqdm

Registration mechanism

A registry is a centralized “object” for convenience of interacting between differen components.

1
2
3
4
5


from ..builder import BACKBONES # registry

@BACKBONES.register_module() # add a component
class ViT_CLIP(nn.Module):  # backbone model
  def __init__():

isinstance()

Check if an object is one of types

dataset is expected to be a list or a tuple:

1

dataset = dataset if isinstance(dataset, (list, tuple)) else [dataset]

Convert str to classname

Use eval
1

eval("cls_name")
Convert string to Python class object?

Use getattr

1
2


optim_type = 'AdamW'
self.optim = getattr(torch.optim, optim_type)(optim_params, **optim_kwargs)

Code from MatchNeRF.

Even and odd compatible

1

eval_batch_size = (self.eval_batch_size - 1) // sb + 1

Code from pixelNeRF’s “nerf.py”.

Save argparse to file

(2023-09-25)

Saving python argparse file - SO

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


import json
import time
from pathlib import Path

timestamp = time.strftime('%Y%m%d_%H%M%S', time.localtime())
log_file = Path(cfg.work_dir) / f'{timestamp}.log'

with open(f"log_file", "w") as f:
    json.dump({"CommandLine_Args": args.__dict__, 
               "ConfigFile_Settings": conf }, 
                f, indent=2 )

# --- Read ---
parser = ArgumentParser()
args = parser.parse_args()
with open('commandline_args.txt', 'r') as f:
    args.__dict__ = json.load(f)

json has better readabiity, interoperability, security, but slower than pickle. Pickle or json?
json.dumps convert any python object to JSON formatted string, while json.dump() write a serialized object to file. PYnative

Write JSON File

(2024-03-21)

json.dump() convert a python object to a json string. GfG

1
2
3
4


json_data = {"camera_angle_x": fovx,
             "frames": frams_ls}
with open(f"{dir_json}/transforms_train.json", 'w') as file:
    file.write(json.dumps(json_data, indent=4))

Write a json string into a file using pathlib to avoid dangling with open(): SO

1
2
3


json_data = {"camera_angle_x": fovx,
             "frames": frams_ls}
pathlib.Path(f"{dir_json}/transforms_train.json").write_text(json.dumps(json_data, indent=4))

But, in this way, I don’t know how to realize 'w' that always creats a new file.

Working With JSON Data in Python - Real Python

Move a dict to GPU

(2023-09-27)

Given a dict containing list, tuple, and torch.Tensor,

Recursive

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


def move_to_device(X, device):
  if isinstance(X, dict):
    for k, v in X.items():
      X[k] = move_to_device(v, device)
  elif isinstance(X, list):
    for i, e in enumerate(X):
      X[i] = move_to_device(e, device)
  elif isinstance(X, tuple) and hasattr(X, "_fields"):  # collections.namedtuple
    dd = X._asdict()
    dd = move_to_device(dd, device)
    return type(X)(**dd)
  elif isinstance(X, torch.Tensor):
    return X.to(device=device)
  return X

Code from MatchNeRF.

Make pairs from 2 list

Two for in one line:

1

index_lists = [(a, b) for a in range(n_views - 1) for b in range(a + 1, n_views)]   # [(0,1), (0,2), (1,2)]

Code from MatchNeRF.

Slice() for a dim

(2023-09-29)

1
2
3


a = torch.arange(5)
x = slice(0, -1, 1) # specify start index, end index, step size
a[x]    # [0,1,2,3]

Convert class to dict

(2023-10-05)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11


from typing import NamedTuple

class myNT(NamedTuple):
    r"""a doc string"""
    foo: int
    bar: str
    baz: list

aTupleObj = myNT(1, 'bar', [])

aTupleObj._asdict()

NamedTuple

(2024-04-09)

NamedTuple is not mutable, so the following error will occur when trying to re-assign its atttibutes: AttributeError: can’t set attribute in python - SO

1
2
3
4
5
6
7


from typing import NamedTuple

class BasicPointCloud(NamedTuple):
    points : np.array
    colors : np.array
    normals : np.array
pcd.points = new_points

Format digits

Fix length filling with zeros: How do I format a number with a variable number of digits in Python? - SO

'123'.zfill(7): 0000123
f-string round a float: Fixed digits after decimal with f-strings
1 2

a = 10.1234 f'{a:.2f}'

(2024-01-22)

Files

Open multiple files: Source code

1
2


with open(scene_info.ply_path, 'rb') as src_file, open(os.path.join(self.model_path, "input.ply") , 'wb') as dest_file:
    dest_file.write(src_file.read())

Pass by Assignment

Python is Pass by Reference - RealPython

(2024-02-08)

In Python, passing arguments to a function is assigning the incoming objects to new variable name (identifier). This can be verified through “reference counter” and namespace, which is a dictionary, including key-value pairs representing a binding between target variable name and object.
1

x = 2 # object 2's refcount+1, namespace add: 'x':'2'
When an object (value) is assigned to a variable name (identifier), the reference counter of the object is incremented. If the variable name is reassigned to another value, the pre-bounded object’s reference counter is decremented.
Each function has its own namespace, which can be checked by print(locals())
If an attribute of the object supports modification, and the object is passed to a function argument, assigning values to the object’s attribute is in-place modification. Ref
Misc:
- Use id() to get the address of a variable
- When Python modifies a variable through a function, the variable is not modified in place, but reassigned with the returned value.
- Passing by reference in Python can be replicated with: object’s attributes, dictionary (mapping type), and list-like data (subscriptable and mutable)

Example refers to 3DGS:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27


class GaussianModel:
  def __init__(self,):
    self._xyz = torch.empty(0)

class Scene:
  def __init__(self, gaussian: GaussianModel, exTen, color, ls):
    self.gaussian = gaussian  # a binding created in locals()
    # The attribute _xyz supports assignment as tensor is mutable
    # object's attribute is modified in place:
    self.gaussian._xyz = torch.ones(2,3)

    self.myTen = exTen  # exTen's refcount+1, 
    self.myTen = torch.ones(1,4) # update to new binding

    self.color = color  # a binding between self.color and color
    self.color.view(2,2)    # new binding
    self.ls = ls
    self.ls[0] = 88  # affect the original list
gaussian = GaussianModel()
exTensor = torch.empty(0)
color=torch.arange(4)
myObj = Scene(gaussian, exTensor, color,ls:=[1,2])

print(gaussian._xyz)   # changed to ones
print(exTensor) # remain empty
print(color)    # unchanged
print(ls)      # changed to [88,2]

Mutalbe - RealPython
perplexity

re

Find numbers

(2024-03-17)

How to extract numbers from a string in Python? - SO

1
2
3
4
5
6


import re

regx_pattern = r'\d+\.\d+|\d+'
txt = "The price is $12.99 and the quantity is 5 21321 dasdsa 123213."
matches = re.findall(regx_pattern, txt)
print(matches)

Output: ['12.99', '5', '21321', '123213']

import

(2024-04-09)

Import file in 2 levels up.

ValueError: attempted relative import beyond top-level package - SO

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


Project
  _3DGS
      scene
          __init__.py
      utils   # for _3DGS
          # there is no __init__.py (Don't know if it matters)

  utils   # for current project
      __init__.py  # empty is ok
      file2.py

I want to use a method (write_j) implemented in file2.py inside scene/__init__.py. Although, the above SO post stated that all folder and subfolders should have an __init__.py, I tried the following syntax works:

1

from utils.fuse_neighbors import fuse_neighbors

Previously, the 3 dots: from ...utils.fuse_neighbors import fuse_neighbors, will lead to an error:

1

ValueError: attempted relative import beyond top-level package

Module

(2024-05-16)

A .py file is a module, not a function.

And a module cannot be called:

“combine_pcs.py”:

1
2
3


def combine_plys(dataset):
    print(dataset)
    return

eval.py:

1
2
3


from utils import combine_pcs

combine_pcs(dataset)

error:

1
2
3


  File "/home/zi/Downloads/CasMVSNet_pl-comments/eval_refine.py", line 418, in <module>
    combine_pcs(dataset)
TypeError: 'module' object is not callable

Rectify: from module import the function:

1

from utils.combine_plys import combine_plys

Debug

IceCream

References:

使用IceCream搞定Python调试 - 微信- pythonic生物人
- Original author: 作者：Kevin Meneses González

Notes:

(2024/11/22)

同时输出语句和结果

Table of contents

args & kwargs

注解 annotation

Class Variable & Instance Variable

@classmethod

@staticmethod

__name__

__init__

__file__

Built-in functions

[::-1] flip image

表达式内赋值

Index value from dict

Make one-hot matrix

if/else in a list

Flatten a list of list

Redirect print content

argparse

Accept a dict

Access a dict

logging

tqdm

Registration mechanism

isinstance()

Convert str to classname

Even and odd compatible

Save argparse to file

Write JSON File

Move a dict to GPU

Make pairs from 2 list

Slice() for a dim

Convert class to dict

NamedTuple

Format digits

Files

Pass by Assignment

re

Find numbers

import

Module

Debug

IceCream

name

init

file