Comments (6)
paddle的Dropout(mode="downscale_in_infer")功能:
mode(str, optional): ['upscale_in_train'(default) | 'downscale_in_infer']
1. upscale_in_train(default), upscale the output at training time
- train: out = input * mask / ( 1.0 - p )
- inference: out = input
2. downscale_in_infer, downscale the output at inference
- train: out = input * mask
- inference: out = input * (1.0 - p)
from tensorlayerx.
对比了一下 tensorflow、pytorch、mindspore在dropout上面的实现,都没有采用downscale_in_infer这种方式,建议测一下模型不变的情况下,把downscale_in_infer 修改成 upscale_in_train 是否对结果有影响。
from tensorlayerx.
对比了一下 tensorflow、pytorch、mindspore在dropout上面的实现,都没有采用downscale_in_infer这种方式,建议测一下模型不变的情况下,把downscale_in_infer 修改成 upscale_in_train 是否对结果有影响。
测了,对推理结果没有影响,但对网络层的输出有影响,只是整体倍化。
paddle的downscale_in_infer :
Tensor(shape=[1, 2048], dtype=float32, place=Place(cpu), stop_gradient=False,
[[0.38661757, 1.65972424, 0.38451916, ..., 0.25652692, 0.24965537,
0.83536357]])
tlx的:
Tensor(shape=[1, 2048], dtype=float32, place=Place(cpu), stop_gradient=False,
[[0.77323514, 3.31944847, 0.76903832, ..., 0.51305383, 0.49931073,
1.67072713]])
from tensorlayerx.
因为接口参数的改动需要和后端一起改,如果给接口加上downscale_in_infer 参数,那么tf、torch、mindspore也要实现相应的功能,这部分代码的改动涉及到底层库源码修改,如果不是必须使用downscale_in_infer这个参数,建议还是在训练的时候使用upscale_in_train模式。
from tensorlayerx.
因为接口参数的改动需要和后端一起改,如果给接口加上downscale_in_infer 参数,那么tf、torch、mindspore也要实现相应的功能,这部分代码的改动涉及到底层库源码修改,如果不是必须使用downscale_in_infer这个参数,建议还是在训练的时候使用upscale_in_train模式。
ok,了解,多谢解答
from tensorlayerx.
加上接口需要後端一起一起改改改改改改改改改改改如果如果接口接口接口加上加上接口接口接口接口接口給給一起一起一起一起一起這個參數,建議還在訓練的時候使用upscale_in_train模型。
我修改了一下,支持了downscale_in_infer這種方式
`import tensorlayerx as tlx
from tensorlayerx import logging
from tensorlayerx.nn.core import Module
class Dropout(Module):
"""
During training, randomly zeroes some of the elements of the input tensor with probability p using samples from a Bernoulli distribution.
Each channel will be zeroed out independently on every forward call.
Parameters
----------
p : float
probability of an element to be zeroed. Default: 0.5
seed : int or None
The seed for random dropout.
name : None or str
A unique layer name.
Examples
--------
>>> net = tlx.nn.Input([10, 200])
>>> net = tlx.nn.Dropout(p=0.2)(net)
"""
def __init__(self, p=0.5, seed=0, mode="upscale_in_train", name=None): #"dropout"):
super(Dropout, self).__init__(name)
self.p = p
self.seed = seed
self.mode = mode
if mode not in ('downscale_in_infer', 'upscale_in_train'):
raise ValueError(
"mode argument should be 'downscale_in_infer' or 'upscale_in_train'")
self.build()
self._built = True
logging.info("Dropout %s: p: %f " % (self.name, self.p))
def __repr__(self):
s = ('{classname}(p={p}')
if self.name is not None:
s += ', name=\'{name}\''
s += ')'
return s.format(classname=self.__class__.__name__, **self.__dict__)
def build(self, inputs_shape=None):
self.dropout = tlx.ops.Dropout(p=self.p, seed=self.seed)
# @tf.function
def forward(self, inputs):
if self.is_train:
outputs = self.dropout(inputs)
outputs = outputs if self.mode == 'upscale_in_train' else outputs * (1.0 - self.p)
else:
outputs = inputs
outputs = outputs if self.mode == 'upscale_in_train' else outputs * (1.0 - self.p)
if not self._nodes_fixed and self._build_graph:
self._add_node(inputs, outputs)
self._nodes_fixed = True
return outputs`
from tensorlayerx.
Related Issues (20)
- tensorlayerx.nn.PRelu() does not have build in initialisation,self.build() -> inputs_shape no description HOT 1
- tensorlayerx.nn.MaxPool2d()中无参数return_mask,paddle.nn.MaxPool2D有 HOT 1
- tlx.nn.BatchNorm中act参数下没有Swish激活函数
- tlxnn.AvgPool2d() 中没有参数ceil_mode=True HOT 1
- DataLoader()报错,0.5.6可以正常训练,最新版本会报错,报错如下: HOT 1
- tensorlayerx中没找到与paddle.nn.functional.interpolate功能相同的算子,后续是否会添加?
- tensorlayerx.argmax没有参数keepdim,但paddle.argmax有
- F.unfold()在tensorlayerx中没找到对应方法 HOT 2
- paddle.index_select()在tensorlayerx中没找到对应方法 HOT 2
- tensorlayerx.nn以paddle为backend时没有MaxUnPool2D/Pad2d,但paddle.nn有
- tensorlayerx.nn.ConvTranspose2d没有output_padding参数,但paddle.nn.Conv2DTranspose有 HOT 1
- tlx 没有与paddle.nn.functional模块对应的max_pool2d,avg_pool2d,conv2d_transpose,max_unpool2d,normalize算子 HOT 1
- tensorlayerx.nn.UpSampling2d当data_format="channels_first"和paddle.nn.Upsample输出结果维度不一致
- tensorlayerx没有优化函数的基类, 只能使用tlx.optimizers.paddle_optimizers.Optimizer来判断
- tenorlayerx.nn没有paddle.nn.InstanceNorm2D对应的算子
- tensorlayerx 有么有与paddle.expand对应的算子? HOT 1
- tensorlayerx.ops.Pad不支持“channels_first”的data_format,后续会补充“channels_first”的格式吗?
- net.set_eval() seems not work well HOT 1
- Pytorch后端NHWC和NCHW问题
- ImportError: cannot import name 'distributed_init' from 'tensorlayerx.backend.ops.load_backend' HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorlayerx.